Note
Click here to download the full example code
Hyperparameter tuning with Ray Tune¶
Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.
Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.
In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.
As you will see, we only need to add some slight modifications. In particular, we need to
- wrap data loading and training in functions,
- make some network parameters configurable,
- add checkpointing (optional),
- and define the search space for the model tuning
To run this tutorial, please make sure the following packages are installed:
ray[tune]: Distributed hyperparameter tuning librarytorchvision: For the data transformers
Setup / Imports¶
Let’s start with the imports:
from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler
Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.
Data loaders¶
We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.
def load_data(data_dir="./data"):
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
trainset = torchvision.datasets.CIFAR10(
root=data_dir, train=True, download=True, transform=transform)
testset = torchvision.datasets.CIFAR10(
root=data_dir, train=False, download=True, transform=transform)
return trainset, testset
Configurable neural network¶
We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:
class Net(nn.Module):
def __init__(self, l1=120, l2=84):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, l1)
self.fc2 = nn.Linear(l1, l2)
self.fc3 = nn.Linear(l2, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
The train function¶
Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.
We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None).
As you can guess, the config parameter will receive the hyperparameters we would like to
train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies
the directory where we load and store the data, so multiple runs can share the same data source.
net = Net(config["l1"], config["l2"])
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
The learning rate of the optimizer is made configurable, too:
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.
Adding (multi) GPU support with DataParallel¶
Image classification benefits largely from GPUs. Luckily, we can continue to use
PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel
to support data parallel training on multiple GPUs:
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
By using a device variable we make sure that training also works when we have
no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly,
like this:
for i, data in enumerate(trainloader, 0):
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.
Communicating with Ray Tune¶
The most interesting part is the communication with Ray Tune:
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.
The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.
Full training function¶
The full code example looks like this:
def train_cifar(config, checkpoint_dir=None, data_dir=None):
net = Net(config["l1"], config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
trainset, testset = load_data(data_dir)
test_abs = int(len(trainset) * 0.8)
train_subset, val_subset = random_split(
trainset, [test_abs, len(trainset) - test_abs])
trainloader = torch.utils.data.DataLoader(
train_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
valloader = torch.utils.data.DataLoader(
val_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
for epoch in range(10): # loop over the dataset multiple times
running_loss = 0.0
epoch_steps = 0
for i, data in enumerate(trainloader, 0):
# get the inputs; data is a list of [inputs, labels]
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
# zero the parameter gradients
optimizer.zero_grad()
# forward + backward + optimize
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
# print statistics
running_loss += loss.item()
epoch_steps += 1
if i % 2000 == 1999: # print every 2000 mini-batches
print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
running_loss / epoch_steps))
running_loss = 0.0
# Validation loss
val_loss = 0.0
val_steps = 0
total = 0
correct = 0
for i, data in enumerate(valloader, 0):
with torch.no_grad():
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
outputs = net(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
loss = criterion(outputs, labels)
val_loss += loss.cpu().numpy()
val_steps += 1
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
print("Finished Training")
As you can see, most of the code is adapted directly from the original example.
Test set accuracy¶
Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:
def test_accuracy(net, device="cpu"):
trainset, testset = load_data()
testloader = torch.utils.data.DataLoader(
testset, batch_size=4, shuffle=False, num_workers=2)
correct = 0
total = 0
with torch.no_grad():
for data in testloader:
images, labels = data
images, labels = images.to(device), labels.to(device)
outputs = net(images)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return correct / total
The function also expects a device parameter, so we can do the
test set validation on a GPU.
Configuring the search space¶
Lastly, we need to define Ray Tune’s search space. Here is an example:
config = {
"l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
The tune.sample_from() function makes it possible to define your own sample
methods to obtain hyperparameters. In this example, the l1 and l2 parameters
should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256.
The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly,
the batch size is a choice between 2, 4, 8, and 16.
At each trial, Ray Tune will now randomly sample a combination of parameters from these
search spaces. It will then train a number of models in parallel and find the best
performing one among these. We also use the ASHAScheduler which will terminate bad
performing trials early.
We wrap the train_cifar function with functools.partial to set the constant
data_dir parameter. We can also tell Ray Tune what resources should be
available for each trial:
gpus_per_trial = 2
# ...
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter,
checkpoint_at_end=True)
You can specify the number of CPUs, which are then available e.g.
to increase the num_workers of the PyTorch DataLoader instances. The selected
number of GPUs are made visible to PyTorch in each trial. Trials do not have access to
GPUs that haven’t been requested for them - so you don’t have to care about two trials
using the same set of resources.
Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is
completely valid. The trials will then share GPUs among each other.
You just have to make sure that the models still fit in the GPU memory.
After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.
The full main function looks like this:
def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
data_dir = os.path.abspath("./data")
load_data(data_dir)
config = {
"l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
scheduler = ASHAScheduler(
metric="loss",
mode="min",
max_t=max_num_epochs,
grace_period=1,
reduction_factor=2)
reporter = CLIReporter(
# parameter_columns=["l1", "l2", "lr", "batch_size"],
metric_columns=["loss", "accuracy", "training_iteration"])
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter)
best_trial = result.get_best_trial("loss", "min", "last")
print("Best trial config: {}".format(best_trial.config))
print("Best trial final validation loss: {}".format(
best_trial.last_result["loss"]))
print("Best trial final validation accuracy: {}".format(
best_trial.last_result["accuracy"]))
best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if gpus_per_trial > 1:
best_trained_model = nn.DataParallel(best_trained_model)
best_trained_model.to(device)
best_checkpoint_dir = best_trial.checkpoint.value
model_state, optimizer_state = torch.load(os.path.join(
best_checkpoint_dir, "checkpoint"))
best_trained_model.load_state_dict(model_state)
test_acc = test_accuracy(best_trained_model, device)
print("Best trial test set accuracy: {}".format(test_acc))
if __name__ == "__main__":
# You can change the number of GPUs per trial here:
main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)
Out:
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz
Extracting /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data
Files already downloaded and verified
== Status ==
Memory usage on this node: 14.3/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_f201f_00000 | RUNNING | | 16 | 128 | 16 | 0.0173062 |
| DEFAULT_f201f_00001 | PENDING | | 2 | 128 | 8 | 0.000447305 |
| DEFAULT_f201f_00002 | PENDING | | 2 | 256 | 256 | 0.0119053 |
| DEFAULT_f201f_00003 | PENDING | | 2 | 32 | 16 | 0.00404483 |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 |
+---------------------+----------+-------+--------------+------+------+-------------+
[2m[36m(pid=5085)[0m Files already downloaded and verified
[2m[36m(pid=5088)[0m Files already downloaded and verified
[2m[36m(pid=5090)[0m Files already downloaded and verified
[2m[36m(pid=5086)[0m Files already downloaded and verified
[2m[36m(pid=5085)[0m Files already downloaded and verified
[2m[36m(pid=5088)[0m Files already downloaded and verified
[2m[36m(pid=5090)[0m Files already downloaded and verified
[2m[36m(pid=5086)[0m Files already downloaded and verified
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_f201f_00000 | RUNNING | | 16 | 128 | 16 | 0.0173062 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 |
+---------------------+----------+-------+--------------+------+------+-------------+
[2m[36m(pid=5086)[0m [1, 2000] loss: 2.144
[2m[36m(pid=5090)[0m [1, 2000] loss: 2.244
[2m[36m(pid=5088)[0m [1, 2000] loss: 2.299
[2m[36m(pid=5085)[0m [1, 2000] loss: 1.794
[2m[36m(pid=5086)[0m [1, 4000] loss: 1.006
[2m[36m(pid=5088)[0m [1, 4000] loss: 1.054
[2m[36m(pid=5090)[0m [1, 4000] loss: 1.110
Result for DEFAULT_f201f_00000:
accuracy: 0.4215
date: 2021-07-29_14-12-50
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.6012202549934387
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 46.481757164001465
time_this_iter_s: 46.481757164001465
time_total_s: 46.481757164001465
timestamp: 1627567970
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.8/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.60122 | 0.4215 | 1 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 | | | |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5086)[0m [1, 6000] loss: 0.648
[2m[36m(pid=5088)[0m [1, 6000] loss: 0.641
[2m[36m(pid=5086)[0m [1, 8000] loss: 0.499
[2m[36m(pid=5090)[0m [1, 6000] loss: 0.720
[2m[36m(pid=5088)[0m [1, 8000] loss: 0.448
[2m[36m(pid=5085)[0m [2, 2000] loss: 1.567
[2m[36m(pid=5088)[0m [1, 10000] loss: 0.347
[2m[36m(pid=5086)[0m [1, 10000] loss: 0.386
Result for DEFAULT_f201f_00000:
accuracy: 0.4254
date: 2021-07-29_14-13-35
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 2
loss: 1.6653225509643554
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 91.32741332054138
time_this_iter_s: 44.84565615653992
time_total_s: 91.32741332054138
timestamp: 1627568015
timesteps_since_restore: 0
training_iteration: 2
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.66532 | 0.4254 | 2 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 | | | |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5090)[0m [1, 8000] loss: 0.539
[2m[36m(pid=5086)[0m [1, 12000] loss: 0.321
[2m[36m(pid=5088)[0m [1, 12000] loss: 0.279
[2m[36m(pid=5085)[0m [3, 2000] loss: 1.525
[2m[36m(pid=5086)[0m [1, 14000] loss: 0.276
[2m[36m(pid=5090)[0m [1, 10000] loss: 0.427
[2m[36m(pid=5088)[0m [1, 14000] loss: 0.234
Result for DEFAULT_f201f_00000:
accuracy: 0.4648
date: 2021-07-29_14-14-16
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 3
loss: 1.5220761643409728
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 132.20206356048584
time_this_iter_s: 40.87465023994446
time_total_s: 132.20206356048584
timestamp: 1627568056
timesteps_since_restore: 0
training_iteration: 3
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.52208 | 0.4648 | 3 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 | | | |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5086)[0m [1, 16000] loss: 0.244
[2m[36m(pid=5090)[0m [1, 12000] loss: 0.357
[2m[36m(pid=5088)[0m [1, 16000] loss: 0.202
[2m[36m(pid=5086)[0m [1, 18000] loss: 0.211
[2m[36m(pid=5085)[0m [4, 2000] loss: 1.498
[2m[36m(pid=5088)[0m [1, 18000] loss: 0.176
Result for DEFAULT_f201f_00000:
accuracy: 0.4372
date: 2021-07-29_14-14-53
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 4
loss: 1.6066401032447815
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 168.9676070213318
time_this_iter_s: 36.76554346084595
time_total_s: 168.9676070213318
timestamp: 1627568093
timesteps_since_restore: 0
training_iteration: 4
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.60664 | 0.4372 | 4 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 | | | |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5090)[0m [1, 14000] loss: 0.314
[2m[36m(pid=5086)[0m [1, 20000] loss: 0.196
[2m[36m(pid=5088)[0m [1, 20000] loss: 0.155
[2m[36m(pid=5090)[0m [1, 16000] loss: 0.279
[2m[36m(pid=5085)[0m [5, 2000] loss: 1.501
Result for DEFAULT_f201f_00000:
accuracy: 0.4442
date: 2021-07-29_14-15-34
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 5
loss: 1.5668423042297364
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 209.80427646636963
time_this_iter_s: 40.83666944503784
time_total_s: 209.80427646636963
timestamp: 1627568134
timesteps_since_restore: 0
training_iteration: 5
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.56684 | 0.4442 | 5 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | | 2 | 32 | 16 | 0.00404483 | | | |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5090)[0m [1, 18000] loss: 0.237
Result for DEFAULT_f201f_00003:
accuracy: 0.2865
date: 2021-07-29_14-15-43
done: true
experiment_id: 8ff027eec3f14b2b9dc5ce8d11408ff8
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.899737369287014
node_ip: 172.17.0.4
pid: 5086
should_checkpoint: true
time_since_restore: 218.76438760757446
time_this_iter_s: 218.76438760757446
time_total_s: 218.76438760757446
timestamp: 1627568143
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00003
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.56684 | 0.4442 | 5 |
| DEFAULT_f201f_00001 | RUNNING | | 2 | 128 | 8 | 0.000447305 | | | |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00003 | RUNNING | 172.17.0.4:5086 | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | PENDING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5083)[0m Files already downloaded and verified
[2m[36m(pid=5083)[0m Files already downloaded and verified
Result for DEFAULT_f201f_00001:
accuracy: 0.4357
date: 2021-07-29_14-15-48
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.541055219617486
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 224.23737812042236
time_this_iter_s: 224.23737812042236
time_total_s: 224.23737812042236
timestamp: 1627568148
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.56684 | 0.4442 | 5 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00004 | RUNNING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5090)[0m [1, 20000] loss: 0.212
[2m[36m(pid=5085)[0m [6, 2000] loss: 1.502
[2m[36m(pid=5083)[0m [1, 2000] loss: 2.252
[2m[36m(pid=5088)[0m [2, 2000] loss: 1.505
Result for DEFAULT_f201f_00000:
accuracy: 0.4722
date: 2021-07-29_14-16-14
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 6
loss: 1.5356089867591858
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 250.69555974006653
time_this_iter_s: 40.8912832736969
time_total_s: 250.69555974006653
timestamp: 1627568174
timesteps_since_restore: 0
training_iteration: 6
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.53561 | 0.4722 | 6 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00002 | RUNNING | | 2 | 256 | 256 | 0.0119053 | | | |
| DEFAULT_f201f_00004 | RUNNING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5083)[0m [1, 4000] loss: 0.963
[2m[36m(pid=5088)[0m [2, 4000] loss: 0.749
Result for DEFAULT_f201f_00002:
accuracy: 0.1789
date: 2021-07-29_14-16-32
done: true
experiment_id: 9a45bdb8086e4379905275a77cbba199
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 2.121293088769913
node_ip: 172.17.0.4
pid: 5090
should_checkpoint: true
time_since_restore: 268.7365050315857
time_this_iter_s: 268.7365050315857
time_total_s: 268.7365050315857
timestamp: 1627568192
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00002
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.53561 | 0.4722 | 6 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00002 | RUNNING | 172.17.0.4:5090 | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00004 | RUNNING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | PENDING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m Files already downloaded and verified
[2m[36m(pid=5087)[0m Files already downloaded and verified
[2m[36m(pid=5083)[0m [1, 6000] loss: 0.580
[2m[36m(pid=5088)[0m [2, 6000] loss: 0.491
[2m[36m(pid=5085)[0m [7, 2000] loss: 1.507
[2m[36m(pid=5083)[0m [1, 8000] loss: 0.414
[2m[36m(pid=5087)[0m [1, 2000] loss: 2.196
Result for DEFAULT_f201f_00000:
accuracy: 0.4847
date: 2021-07-29_14-17-00
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 7
loss: 1.5526806345939637
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 296.0717935562134
time_this_iter_s: 45.37623381614685
time_total_s: 296.0717935562134
timestamp: 1627568220
timesteps_since_restore: 0
training_iteration: 7
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.55268 | 0.4847 | 7 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00004 | RUNNING | | 4 | 8 | 16 | 0.00134526 | | | |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [2, 8000] loss: 0.365
[2m[36m(pid=5083)[0m [1, 10000] loss: 0.318
[2m[36m(pid=5087)[0m [1, 4000] loss: 0.984
[2m[36m(pid=5088)[0m [2, 10000] loss: 0.292
[2m[36m(pid=5085)[0m [8, 2000] loss: 1.514
Result for DEFAULT_f201f_00004:
accuracy: 0.3837
date: 2021-07-29_14-17-31
done: false
experiment_id: 29b8f76bd84947dda7dcc9751ba1012d
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.6751174277424812
node_ip: 172.17.0.4
pid: 5083
should_checkpoint: true
time_since_restore: 106.32377362251282
time_this_iter_s: 106.32377362251282
time_total_s: 106.32377362251282
timestamp: 1627568251
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00004
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.55268 | 0.4847 | 7 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00004 | RUNNING | 172.17.0.4:5083 | 4 | 8 | 16 | 0.00134526 | 1.67512 | 0.3837 | 1 |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m [1, 6000] loss: 0.603
[2m[36m(pid=5088)[0m [2, 12000] loss: 0.236
Result for DEFAULT_f201f_00000:
accuracy: 0.4122
date: 2021-07-29_14-17-41
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 8
loss: 1.6352525778770446
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 336.84724020957947
time_this_iter_s: 40.77544665336609
time_total_s: 336.84724020957947
timestamp: 1627568261
timesteps_since_restore: 0
training_iteration: 8
trial_id: f201f_00000
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.63525 | 0.4122 | 8 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00004 | RUNNING | 172.17.0.4:5083 | 4 | 8 | 16 | 0.00134526 | 1.67512 | 0.3837 | 1 |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m [1, 8000] loss: 0.435
[2m[36m(pid=5083)[0m [2, 2000] loss: 1.556
[2m[36m(pid=5088)[0m [2, 14000] loss: 0.202
[2m[36m(pid=5087)[0m [1, 10000] loss: 0.343
[2m[36m(pid=5085)[0m [9, 2000] loss: 1.508
[2m[36m(pid=5088)[0m [2, 16000] loss: 0.173
[2m[36m(pid=5083)[0m [2, 4000] loss: 0.756
Result for DEFAULT_f201f_00000:
accuracy: 0.4686
date: 2021-07-29_14-18-23
done: false
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 9
loss: 1.5646762167930603
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 378.9824676513672
time_this_iter_s: 42.13522744178772
time_total_s: 378.9824676513672
timestamp: 1627568303
timesteps_since_restore: 0
training_iteration: 9
trial_id: f201f_00000
== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.56468 | 0.4686 | 9 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00004 | RUNNING | 172.17.0.4:5083 | 4 | 8 | 16 | 0.00134526 | 1.67512 | 0.3837 | 1 |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m [1, 12000] loss: 0.276
[2m[36m(pid=5083)[0m [2, 6000] loss: 0.508
[2m[36m(pid=5088)[0m [2, 18000] loss: 0.153
[2m[36m(pid=5087)[0m [1, 14000] loss: 0.231
[2m[36m(pid=5083)[0m [2, 8000] loss: 0.368
[2m[36m(pid=5085)[0m [10, 2000] loss: 1.511
[2m[36m(pid=5088)[0m [2, 20000] loss: 0.138
Result for DEFAULT_f201f_00000:
accuracy: 0.4533
date: 2021-07-29_14-19-04
done: true
experiment_id: 519e73cc629f4a16ad544ba419c613fd
hostname: 9ec45cf3ba83
iterations_since_restore: 10
loss: 1.6734770732879638
node_ip: 172.17.0.4
pid: 5085
should_checkpoint: true
time_since_restore: 420.25495433807373
time_this_iter_s: 41.27248668670654
time_total_s: 420.25495433807373
timestamp: 1627568344
timesteps_since_restore: 0
training_iteration: 10
trial_id: f201f_00000
== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING | 172.17.0.4:5085 | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.54106 | 0.4357 | 1 |
| DEFAULT_f201f_00004 | RUNNING | 172.17.0.4:5083 | 4 | 8 | 16 | 0.00134526 | 1.67512 | 0.3837 | 1 |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | PENDING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m [1, 16000] loss: 0.201
[2m[36m(pid=5089)[0m Files already downloaded and verified
[2m[36m(pid=5089)[0m Files already downloaded and verified
[2m[36m(pid=5083)[0m [2, 10000] loss: 0.296
[2m[36m(pid=5087)[0m [1, 18000] loss: 0.177
Result for DEFAULT_f201f_00001:
accuracy: 0.4878
date: 2021-07-29_14-19-25
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 2
loss: 1.4001507631493733
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 441.5508849620819
time_this_iter_s: 217.31350684165955
time_total_s: 441.5508849620819
timestamp: 1627568365
timesteps_since_restore: 0
training_iteration: 2
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5327366570568643 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.40015 | 0.4878 | 2 |
| DEFAULT_f201f_00004 | RUNNING | 172.17.0.4:5083 | 4 | 8 | 16 | 0.00134526 | 1.67512 | 0.3837 | 1 |
| DEFAULT_f201f_00005 | RUNNING | | 2 | 16 | 16 | 0.00126487 | | | |
| DEFAULT_f201f_00006 | RUNNING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | PENDING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_f201f_00004:
accuracy: 0.4063
date: 2021-07-29_14-19-28
done: true
experiment_id: 29b8f76bd84947dda7dcc9751ba1012d
hostname: 9ec45cf3ba83
iterations_since_restore: 2
loss: 1.6531679716050625
node_ip: 172.17.0.4
pid: 5083
should_checkpoint: true
time_since_restore: 223.1200053691864
time_this_iter_s: 116.79623174667358
time_total_s: 223.1200053691864
timestamp: 1627568368
timesteps_since_restore: 0
training_iteration: 2
trial_id: f201f_00004
[2m[36m(pid=5084)[0m Files already downloaded and verified
[2m[36m(pid=5089)[0m [1, 2000] loss: 2.342
[2m[36m(pid=5084)[0m Files already downloaded and verified
[2m[36m(pid=5087)[0m [1, 20000] loss: 0.157
[2m[36m(pid=5088)[0m [3, 2000] loss: 1.328
[2m[36m(pid=5089)[0m [1, 4000] loss: 1.175
[2m[36m(pid=5084)[0m [1, 2000] loss: 2.300
[2m[36m(pid=5088)[0m [3, 4000] loss: 0.666
[2m[36m(pid=5089)[0m [1, 6000] loss: 0.782
[2m[36m(pid=5084)[0m [1, 4000] loss: 1.112
Result for DEFAULT_f201f_00005:
accuracy: 0.4437
date: 2021-07-29_14-20-13
done: false
experiment_id: 0aba26e190ca42cba089c32a9006ff74
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.5085618595972656
node_ip: 172.17.0.4
pid: 5087
should_checkpoint: true
time_since_restore: 218.37232851982117
time_this_iter_s: 218.37232851982117
time_total_s: 218.37232851982117
timestamp: 1627568413
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00005
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.63816884136796
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.40015 | 0.4878 | 2 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.50856 | 0.4437 | 1 |
| DEFAULT_f201f_00006 | RUNNING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | RUNNING | | 8 | 64 | 256 | 0.000285913 | | | |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [3, 6000] loss: 0.432
[2m[36m(pid=5089)[0m [1, 8000] loss: 0.587
[2m[36m(pid=5087)[0m [2, 2000] loss: 1.513
Result for DEFAULT_f201f_00007:
accuracy: 0.2903
date: 2021-07-29_14-20-35
done: true
experiment_id: b17549cee3d041f19ecfddc572127732
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.9859605166435241
node_ip: 172.17.0.4
pid: 5084
should_checkpoint: true
time_since_restore: 65.69195485115051
time_this_iter_s: 65.69195485115051
time_total_s: 65.69195485115051
timestamp: 1627568435
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00007
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.40015 | 0.4878 | 2 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.50856 | 0.4437 | 1 |
| DEFAULT_f201f_00006 | RUNNING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00007 | RUNNING | 172.17.0.4:5084 | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | PENDING | | 8 | 32 | 16 | 0.00172471 | | | |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [3, 8000] loss: 0.324
[2m[36m(pid=8726)[0m Files already downloaded and verified
[2m[36m(pid=8726)[0m Files already downloaded and verified
[2m[36m(pid=5089)[0m [1, 10000] loss: 0.469
[2m[36m(pid=5087)[0m [2, 4000] loss: 0.768
[2m[36m(pid=5088)[0m [3, 10000] loss: 0.257
[2m[36m(pid=5089)[0m [1, 12000] loss: 0.392
[2m[36m(pid=8726)[0m [1, 2000] loss: 2.275
[2m[36m(pid=5087)[0m [2, 6000] loss: 0.506
[2m[36m(pid=5088)[0m [3, 12000] loss: 0.218
[2m[36m(pid=5087)[0m [2, 8000] loss: 0.377
[2m[36m(pid=8726)[0m [1, 4000] loss: 0.945
[2m[36m(pid=5089)[0m [1, 14000] loss: 0.336
[2m[36m(pid=5088)[0m [3, 14000] loss: 0.182
[2m[36m(pid=5087)[0m [2, 10000] loss: 0.303
[2m[36m(pid=5088)[0m [3, 16000] loss: 0.158
Result for DEFAULT_f201f_00008:
accuracy: 0.4047
date: 2021-07-29_14-21-45
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 1.620893050098419
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 66.44279766082764
time_this_iter_s: 66.44279766082764
time_total_s: 66.44279766082764
timestamp: 1627568505
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.64800523892045
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.40015 | 0.4878 | 2 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.50856 | 0.4437 | 1 |
| DEFAULT_f201f_00006 | RUNNING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.62089 | 0.4047 | 1 |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5089)[0m [1, 16000] loss: 0.293
[2m[36m(pid=5087)[0m [2, 12000] loss: 0.250
[2m[36m(pid=5088)[0m [3, 18000] loss: 0.142
[2m[36m(pid=5087)[0m [2, 14000] loss: 0.210
[2m[36m(pid=8726)[0m [2, 2000] loss: 1.533
[2m[36m(pid=5089)[0m [1, 18000] loss: 0.261
[2m[36m(pid=5088)[0m [3, 20000] loss: 0.130
[2m[36m(pid=5087)[0m [2, 16000] loss: 0.189
[2m[36m(pid=5089)[0m [1, 20000] loss: 0.235
[2m[36m(pid=8726)[0m [2, 4000] loss: 0.731
[2m[36m(pid=5087)[0m [2, 18000] loss: 0.167
Result for DEFAULT_f201f_00008:
accuracy: 0.4996
date: 2021-07-29_14-22-48
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 2
loss: 1.3640813674926757
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 129.95181798934937
time_this_iter_s: 63.50902032852173
time_total_s: 129.95181798934937
timestamp: 1627568568
timesteps_since_restore: 0
training_iteration: 2
trial_id: f201f_00008
== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.526659367377218 | Iter 1.000: -1.64800523892045
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.40015 | 0.4878 | 2 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.50856 | 0.4437 | 1 |
| DEFAULT_f201f_00006 | RUNNING | | 2 | 256 | 16 | 0.0404286 | | | |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.36408 | 0.4996 | 2 |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5087)[0m [2, 20000] loss: 0.151
Result for DEFAULT_f201f_00001:
accuracy: 0.5397
date: 2021-07-29_14-22-53
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 3
loss: 1.2915692168400623
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 649.3570704460144
time_this_iter_s: 207.8061854839325
time_total_s: 649.3570704460144
timestamp: 1627568573
timesteps_since_restore: 0
training_iteration: 3
trial_id: f201f_00001
Result for DEFAULT_f201f_00006:
accuracy: 0.0956
date: 2021-07-29_14-23-00
done: true
experiment_id: 7dc069aa9c2144ee9e8e492ed6b3f78a
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 2.3285563125133515
node_ip: 172.17.0.4
pid: 5089
should_checkpoint: true
time_since_restore: 233.9469187259674
time_this_iter_s: 233.9469187259674
time_total_s: 233.9469187259674
timestamp: 1627568580
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00006
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.526659367377218 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.50856 | 0.4437 | 1 |
| DEFAULT_f201f_00006 | RUNNING | 172.17.0.4:5089 | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.36408 | 0.4996 | 2 |
| DEFAULT_f201f_00009 | PENDING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=9546)[0m Files already downloaded and verified
[2m[36m(pid=9546)[0m Files already downloaded and verified
[2m[36m(pid=8726)[0m [3, 2000] loss: 1.361
[2m[36m(pid=5088)[0m [4, 2000] loss: 1.217
Result for DEFAULT_f201f_00005:
accuracy: 0.4488
date: 2021-07-29_14-23-27
done: true
experiment_id: 0aba26e190ca42cba089c32a9006ff74
hostname: 9ec45cf3ba83
iterations_since_restore: 2
loss: 1.5376347200442106
node_ip: 172.17.0.4
pid: 5087
should_checkpoint: true
time_since_restore: 411.8736696243286
time_this_iter_s: 193.50134110450745
time_total_s: 411.8736696243286
timestamp: 1627568607
timesteps_since_restore: 0
training_iteration: 2
trial_id: f201f_00005
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00005 | RUNNING | 172.17.0.4:5087 | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.36408 | 0.4996 | 2 |
| DEFAULT_f201f_00009 | RUNNING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [4, 4000] loss: 0.596
[2m[36m(pid=9546)[0m [1, 2000] loss: 2.319
[2m[36m(pid=8726)[0m [3, 4000] loss: 0.668
[2m[36m(pid=5088)[0m [4, 6000] loss: 0.405
[2m[36m(pid=9546)[0m [1, 4000] loss: 1.160
Result for DEFAULT_f201f_00008:
accuracy: 0.529
date: 2021-07-29_14-23-50
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 3
loss: 1.320628886294365
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 192.0822982788086
time_this_iter_s: 62.13048028945923
time_total_s: 192.0822982788086
timestamp: 1627568630
timesteps_since_restore: 0
training_iteration: 3
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.4/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.6751174277424812
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.32063 | 0.529 | 3 |
| DEFAULT_f201f_00009 | RUNNING | | 8 | 64 | 16 | 0.0574224 | | | |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [4, 8000] loss: 0.297
Result for DEFAULT_f201f_00009:
accuracy: 0.0983
date: 2021-07-29_14-24-06
done: true
experiment_id: e3e77385e2d9496db44502e89f7563e3
hostname: 9ec45cf3ba83
iterations_since_restore: 1
loss: 2.319805237197876
node_ip: 172.17.0.4
pid: 9546
should_checkpoint: true
time_since_restore: 64.11058259010315
time_this_iter_s: 64.11058259010315
time_total_s: 64.11058259010315
timestamp: 1627568646
timesteps_since_restore: 0
training_iteration: 1
trial_id: f201f_00009
== Status ==
Memory usage on this node: 15.4/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.32063 | 0.529 | 3 |
| DEFAULT_f201f_00009 | RUNNING | 172.17.0.4:9546 | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=8726)[0m [4, 2000] loss: 1.258
[2m[36m(pid=5088)[0m [4, 10000] loss: 0.241
[2m[36m(pid=8726)[0m [4, 4000] loss: 0.631
[2m[36m(pid=5088)[0m [4, 12000] loss: 0.197
Result for DEFAULT_f201f_00008:
accuracy: 0.5519
date: 2021-07-29_14-24-40
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 4
loss: 1.273954435968399
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 242.25610089302063
time_this_iter_s: 50.173802614212036
time_total_s: 242.25610089302063
timestamp: 1627568680
timesteps_since_restore: 0
training_iteration: 4
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.44029726960659 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.27395 | 0.5519 | 4 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [4, 14000] loss: 0.170
[2m[36m(pid=8726)[0m [5, 2000] loss: 1.193
[2m[36m(pid=5088)[0m [4, 16000] loss: 0.152
[2m[36m(pid=8726)[0m [5, 4000] loss: 0.607
[2m[36m(pid=5088)[0m [4, 18000] loss: 0.134
Result for DEFAULT_f201f_00008:
accuracy: 0.5585
date: 2021-07-29_14-25-34
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 5
loss: 1.2658138296961785
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 296.21075320243835
time_this_iter_s: 53.954652309417725
time_total_s: 296.21075320243835
timestamp: 1627568734
timesteps_since_restore: 0
training_iteration: 5
trial_id: f201f_00008
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.44029726960659 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.29157 | 0.5397 | 3 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.26581 | 0.5585 | 5 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [4, 20000] loss: 0.117
[2m[36m(pid=8726)[0m [6, 2000] loss: 1.159
[2m[36m(pid=8726)[0m [6, 4000] loss: 0.583
Result for DEFAULT_f201f_00001:
accuracy: 0.5831
date: 2021-07-29_14-26-17
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 4
loss: 1.1855550648670643
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 853.2846131324768
time_this_iter_s: 203.9275426864624
time_total_s: 853.2846131324768
timestamp: 1627568777
timesteps_since_restore: 0
training_iteration: 4
trial_id: f201f_00001
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18556 | 0.5831 | 4 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.26581 | 0.5585 | 5 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_f201f_00008:
accuracy: 0.5757
date: 2021-07-29_14-26-31
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 6
loss: 1.213883140027523
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 352.9563558101654
time_this_iter_s: 56.74560260772705
time_total_s: 352.9563558101654
timestamp: 1627568791
timesteps_since_restore: 0
training_iteration: 6
trial_id: f201f_00008
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18556 | 0.5831 | 4 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.21388 | 0.5757 | 6 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [5, 2000] loss: 1.107
[2m[36m(pid=5088)[0m [5, 4000] loss: 0.575
[2m[36m(pid=8726)[0m [7, 2000] loss: 1.115
[2m[36m(pid=5088)[0m [5, 6000] loss: 0.372
[2m[36m(pid=8726)[0m [7, 4000] loss: 0.568
[2m[36m(pid=5088)[0m [5, 8000] loss: 0.280
Result for DEFAULT_f201f_00008:
accuracy: 0.5837
date: 2021-07-29_14-27-20
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 7
loss: 1.1828787779331207
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 401.7406098842621
time_this_iter_s: 48.78425407409668
time_total_s: 401.7406098842621
timestamp: 1627568840
timesteps_since_restore: 0
training_iteration: 7
trial_id: f201f_00008
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18556 | 0.5831 | 4 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.18288 | 0.5837 | 7 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [5, 10000] loss: 0.230
[2m[36m(pid=8726)[0m [8, 2000] loss: 1.091
[2m[36m(pid=5088)[0m [5, 12000] loss: 0.187
[2m[36m(pid=5088)[0m [5, 14000] loss: 0.159
[2m[36m(pid=8726)[0m [8, 4000] loss: 0.553
[2m[36m(pid=5088)[0m [5, 16000] loss: 0.141
Result for DEFAULT_f201f_00008:
accuracy: 0.5696
date: 2021-07-29_14-28-16
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 8
loss: 1.2416244268774985
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 458.10355401039124
time_this_iter_s: 56.36294412612915
time_total_s: 458.10355401039124
timestamp: 1627568896
timesteps_since_restore: 0
training_iteration: 8
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18556 | 0.5831 | 4 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.24162 | 0.5696 | 8 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [5, 18000] loss: 0.127
[2m[36m(pid=8726)[0m [9, 2000] loss: 1.074
[2m[36m(pid=5088)[0m [5, 20000] loss: 0.113
[2m[36m(pid=8726)[0m [9, 4000] loss: 0.545
Result for DEFAULT_f201f_00008:
accuracy: 0.5852
date: 2021-07-29_14-29-06
done: false
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 9
loss: 1.1980089530587197
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 508.1109640598297
time_this_iter_s: 50.00741004943848
time_total_s: 508.1109640598297
timestamp: 1627568946
timesteps_since_restore: 0
training_iteration: 9
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18556 | 0.5831 | 4 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.19801 | 0.5852 | 9 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_f201f_00001:
accuracy: 0.5986
date: 2021-07-29_14-29-13
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 5
loss: 1.15476273704404
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1029.5660145282745
time_this_iter_s: 176.28140139579773
time_total_s: 1029.5660145282745
timestamp: 1627568953
timesteps_since_restore: 0
training_iteration: 5
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.15476 | 0.5986 | 5 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.19801 | 0.5852 | 9 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=8726)[0m [10, 2000] loss: 1.034
[2m[36m(pid=5088)[0m [6, 2000] loss: 1.071
[2m[36m(pid=8726)[0m [10, 4000] loss: 0.538
[2m[36m(pid=5088)[0m [6, 4000] loss: 0.515
Result for DEFAULT_f201f_00008:
accuracy: 0.5787
date: 2021-07-29_14-29-53
done: true
experiment_id: 828c343465b746a6833f7a13f52e8c21
hostname: 9ec45cf3ba83
iterations_since_restore: 10
loss: 1.2242828347325325
node_ip: 172.17.0.4
pid: 8726
should_checkpoint: true
time_since_restore: 554.6373965740204
time_this_iter_s: 46.526432514190674
time_total_s: 554.6373965740204
timestamp: 1627568993
timesteps_since_restore: 0
training_iteration: 10
trial_id: f201f_00008
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.15476 | 0.5986 | 5 |
| DEFAULT_f201f_00008 | RUNNING | 172.17.0.4:8726 | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [6, 6000] loss: 0.354
[2m[36m(pid=5088)[0m [6, 8000] loss: 0.259
[2m[36m(pid=5088)[0m [6, 10000] loss: 0.218
[2m[36m(pid=5088)[0m [6, 12000] loss: 0.179
[2m[36m(pid=5088)[0m [6, 14000] loss: 0.156
[2m[36m(pid=5088)[0m [6, 16000] loss: 0.136
[2m[36m(pid=5088)[0m [6, 18000] loss: 0.116
[2m[36m(pid=5088)[0m [6, 20000] loss: 0.108
Result for DEFAULT_f201f_00001:
accuracy: 0.5862
date: 2021-07-29_14-31-07
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 6
loss: 1.1807675102452049
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1143.6345767974854
time_this_iter_s: 114.06856226921082
time_total_s: 1143.6345767974854
timestamp: 1627569067
timesteps_since_restore: 0
training_iteration: 6
trial_id: f201f_00001
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.18077 | 0.5862 | 6 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [7, 2000] loss: 0.969
[2m[36m(pid=5088)[0m [7, 4000] loss: 0.489
[2m[36m(pid=5088)[0m [7, 6000] loss: 0.332
[2m[36m(pid=5088)[0m [7, 8000] loss: 0.258
[2m[36m(pid=5088)[0m [7, 10000] loss: 0.207
[2m[36m(pid=5088)[0m [7, 12000] loss: 0.170
[2m[36m(pid=5088)[0m [7, 14000] loss: 0.144
[2m[36m(pid=5088)[0m [7, 16000] loss: 0.130
[2m[36m(pid=5088)[0m [7, 18000] loss: 0.114
[2m[36m(pid=5088)[0m [7, 20000] loss: 0.104
Result for DEFAULT_f201f_00001:
accuracy: 0.5818
date: 2021-07-29_14-32-42
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 7
loss: 1.220972312376625
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1238.2733232975006
time_this_iter_s: 94.63874650001526
time_total_s: 1238.2733232975006
timestamp: 1627569162
timesteps_since_restore: 0
training_iteration: 7
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.22097 | 0.5818 | 7 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [8, 2000] loss: 0.954
[2m[36m(pid=5088)[0m [8, 4000] loss: 0.465
[2m[36m(pid=5088)[0m [8, 6000] loss: 0.314
[2m[36m(pid=5088)[0m [8, 8000] loss: 0.246
[2m[36m(pid=5088)[0m [8, 10000] loss: 0.196
[2m[36m(pid=5088)[0m [8, 12000] loss: 0.161
[2m[36m(pid=5088)[0m [8, 14000] loss: 0.142
[2m[36m(pid=5088)[0m [8, 16000] loss: 0.124
[2m[36m(pid=5088)[0m [8, 18000] loss: 0.108
[2m[36m(pid=5088)[0m [8, 20000] loss: 0.098
Result for DEFAULT_f201f_00001:
accuracy: 0.5836
date: 2021-07-29_14-34-13
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 8
loss: 1.246836426538159
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1329.7241804599762
time_this_iter_s: 91.45085716247559
time_total_s: 1329.7241804599762
timestamp: 1627569253
timesteps_since_restore: 0
training_iteration: 8
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.24684 | 0.5836 | 8 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [9, 2000] loss: 0.924
[2m[36m(pid=5088)[0m [9, 4000] loss: 0.440
[2m[36m(pid=5088)[0m [9, 6000] loss: 0.298
[2m[36m(pid=5088)[0m [9, 8000] loss: 0.228
[2m[36m(pid=5088)[0m [9, 10000] loss: 0.183
[2m[36m(pid=5088)[0m [9, 12000] loss: 0.158
[2m[36m(pid=5088)[0m [9, 14000] loss: 0.138
[2m[36m(pid=5088)[0m [9, 16000] loss: 0.117
[2m[36m(pid=5088)[0m [9, 18000] loss: 0.110
[2m[36m(pid=5088)[0m [9, 20000] loss: 0.096
Result for DEFAULT_f201f_00001:
accuracy: 0.5968
date: 2021-07-29_14-35-49
done: false
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 9
loss: 1.2085561733988057
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1425.0174763202667
time_this_iter_s: 95.29329586029053
time_total_s: 1425.0174763202667
timestamp: 1627569349
timesteps_since_restore: 0
training_iteration: 9
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.20856 | 0.5968 | 9 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=5088)[0m [10, 2000] loss: 0.828
[2m[36m(pid=5088)[0m [10, 4000] loss: 0.427
[2m[36m(pid=5088)[0m [10, 6000] loss: 0.290
[2m[36m(pid=5088)[0m [10, 8000] loss: 0.225
[2m[36m(pid=5088)[0m [10, 10000] loss: 0.176
[2m[36m(pid=5088)[0m [10, 12000] loss: 0.152
[2m[36m(pid=5088)[0m [10, 14000] loss: 0.131
[2m[36m(pid=5088)[0m [10, 16000] loss: 0.115
[2m[36m(pid=5088)[0m [10, 18000] loss: 0.102
[2m[36m(pid=5088)[0m [10, 20000] loss: 0.091
Result for DEFAULT_f201f_00001:
accuracy: 0.5926
date: 2021-07-29_14-37-26
done: true
experiment_id: 3f191cc7a1ef496990f3537841cc639f
hostname: 9ec45cf3ba83
iterations_since_restore: 10
loss: 1.23822381478647
node_ip: 172.17.0.4
pid: 5088
should_checkpoint: true
time_since_restore: 1522.388427734375
time_this_iter_s: 97.37095141410828
time_total_s: 1522.388427734375
timestamp: 1627569446
timesteps_since_restore: 0
training_iteration: 10
trial_id: f201f_00001
== Status ==
Memory usage on this node: 15.1/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING | 172.17.0.4:5088 | 2 | 128 | 8 | 0.000447305 | 1.23822 | 0.5926 | 10 |
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | TERMINATED | | 16 | 128 | 16 | 0.0173062 | 1.67348 | 0.4533 | 10 |
| DEFAULT_f201f_00001 | TERMINATED | | 2 | 128 | 8 | 0.000447305 | 1.23822 | 0.5926 | 10 |
| DEFAULT_f201f_00002 | TERMINATED | | 2 | 256 | 256 | 0.0119053 | 2.12129 | 0.1789 | 1 |
| DEFAULT_f201f_00003 | TERMINATED | | 2 | 32 | 16 | 0.00404483 | 1.89974 | 0.2865 | 1 |
| DEFAULT_f201f_00004 | TERMINATED | | 4 | 8 | 16 | 0.00134526 | 1.65317 | 0.4063 | 2 |
| DEFAULT_f201f_00005 | TERMINATED | | 2 | 16 | 16 | 0.00126487 | 1.53763 | 0.4488 | 2 |
| DEFAULT_f201f_00006 | TERMINATED | | 2 | 256 | 16 | 0.0404286 | 2.32856 | 0.0956 | 1 |
| DEFAULT_f201f_00007 | TERMINATED | | 8 | 64 | 256 | 0.000285913 | 1.98596 | 0.2903 | 1 |
| DEFAULT_f201f_00008 | TERMINATED | | 8 | 32 | 16 | 0.00172471 | 1.22428 | 0.5787 | 10 |
| DEFAULT_f201f_00009 | TERMINATED | | 8 | 64 | 16 | 0.0574224 | 2.31981 | 0.0983 | 1 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
Best trial config: {'l1': 32, 'l2': 16, 'lr': 0.001724709016268422, 'batch_size': 8}
Best trial final validation loss: 1.2242828347325325
Best trial final validation accuracy: 0.5787
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.5797
If you run the code, an example output could look like this:
Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... | l1 | l2 | lr | batch_size | loss | accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... | 64 | 4 | 0.00011629 | 2 | 1.87273 | 0.244 | 2 |
| ... | 32 | 64 | 0.000339763 | 8 | 1.23603 | 0.567 | 8 |
| ... | 8 | 16 | 0.00276249 | 16 | 1.1815 | 0.5836 | 10 |
| ... | 4 | 64 | 0.000648721 | 4 | 1.31131 | 0.5224 | 8 |
| ... | 32 | 16 | 0.000340753 | 8 | 1.26454 | 0.5444 | 8 |
| ... | 8 | 4 | 0.000699775 | 8 | 1.99594 | 0.1983 | 2 |
| ... | 256 | 8 | 0.0839654 | 16 | 2.3119 | 0.0993 | 1 |
| ... | 16 | 128 | 0.0758154 | 16 | 2.33575 | 0.1327 | 1 |
| ... | 16 | 8 | 0.0763312 | 16 | 2.31129 | 0.1042 | 4 |
| ... | 128 | 16 | 0.000124903 | 4 | 2.26917 | 0.1945 | 1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+
Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806
Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.
So that’s it! You can now tune the parameters of your PyTorch models.
Total running time of the script: ( 26 minutes 23.448 seconds)