Note
Click here to download the full example code
Hyperparameter tuning with Ray Tune¶
Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.
Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.
In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.
As you will see, we only need to add some slight modifications. In particular, we need to
wrap data loading and training in functions,
make some network parameters configurable,
add checkpointing (optional),
and define the search space for the model tuning
To run this tutorial, please make sure the following packages are installed:
ray[tune]
: Distributed hyperparameter tuning librarytorchvision
: For the data transformers
Setup / Imports¶
Let’s start with the imports:
from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler
Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.
Data loaders¶
We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.
def load_data(data_dir="./data"):
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
trainset = torchvision.datasets.CIFAR10(
root=data_dir, train=True, download=True, transform=transform)
testset = torchvision.datasets.CIFAR10(
root=data_dir, train=False, download=True, transform=transform)
return trainset, testset
Configurable neural network¶
We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:
class Net(nn.Module):
def __init__(self, l1=120, l2=84):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, l1)
self.fc2 = nn.Linear(l1, l2)
self.fc3 = nn.Linear(l2, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
The train function¶
Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.
We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None)
.
As you can guess, the config
parameter will receive the hyperparameters we would like to
train with. The checkpoint_dir
parameter is used to restore checkpoints. The data_dir
specifies
the directory where we load and store the data, so multiple runs can share the same data source.
net = Net(config["l1"], config["l2"])
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
The learning rate of the optimizer is made configurable, too:
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.
Adding (multi) GPU support with DataParallel¶
Image classification benefits largely from GPUs. Luckily, we can continue to use
PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel
to support data parallel training on multiple GPUs:
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
By using a device
variable we make sure that training also works when we have
no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly,
like this:
for i, data in enumerate(trainloader, 0):
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.
Communicating with Ray Tune¶
The most interesting part is the communication with Ray Tune:
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.
The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.
Full training function¶
The full code example looks like this:
def train_cifar(config, checkpoint_dir=None, data_dir=None):
net = Net(config["l1"], config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
trainset, testset = load_data(data_dir)
test_abs = int(len(trainset) * 0.8)
train_subset, val_subset = random_split(
trainset, [test_abs, len(trainset) - test_abs])
trainloader = torch.utils.data.DataLoader(
train_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
valloader = torch.utils.data.DataLoader(
val_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
for epoch in range(10): # loop over the dataset multiple times
running_loss = 0.0
epoch_steps = 0
for i, data in enumerate(trainloader, 0):
# get the inputs; data is a list of [inputs, labels]
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
# zero the parameter gradients
optimizer.zero_grad()
# forward + backward + optimize
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
# print statistics
running_loss += loss.item()
epoch_steps += 1
if i % 2000 == 1999: # print every 2000 mini-batches
print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
running_loss / epoch_steps))
running_loss = 0.0
# Validation loss
val_loss = 0.0
val_steps = 0
total = 0
correct = 0
for i, data in enumerate(valloader, 0):
with torch.no_grad():
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
outputs = net(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
loss = criterion(outputs, labels)
val_loss += loss.cpu().numpy()
val_steps += 1
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
print("Finished Training")
As you can see, most of the code is adapted directly from the original example.
Test set accuracy¶
Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:
def test_accuracy(net, device="cpu"):
trainset, testset = load_data()
testloader = torch.utils.data.DataLoader(
testset, batch_size=4, shuffle=False, num_workers=2)
correct = 0
total = 0
with torch.no_grad():
for data in testloader:
images, labels = data
images, labels = images.to(device), labels.to(device)
outputs = net(images)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return correct / total
The function also expects a device
parameter, so we can do the
test set validation on a GPU.
Configuring the search space¶
Lastly, we need to define Ray Tune’s search space. Here is an example:
config = {
"l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
The tune.sample_from()
function makes it possible to define your own sample
methods to obtain hyperparameters. In this example, the l1
and l2
parameters
should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256.
The lr
(learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly,
the batch size is a choice between 2, 4, 8, and 16.
At each trial, Ray Tune will now randomly sample a combination of parameters from these
search spaces. It will then train a number of models in parallel and find the best
performing one among these. We also use the ASHAScheduler
which will terminate bad
performing trials early.
We wrap the train_cifar
function with functools.partial
to set the constant
data_dir
parameter. We can also tell Ray Tune what resources should be
available for each trial:
gpus_per_trial = 2
# ...
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter,
checkpoint_at_end=True)
You can specify the number of CPUs, which are then available e.g.
to increase the num_workers
of the PyTorch DataLoader
instances. The selected
number of GPUs are made visible to PyTorch in each trial. Trials do not have access to
GPUs that haven’t been requested for them - so you don’t have to care about two trials
using the same set of resources.
Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5
is
completely valid. The trials will then share GPUs among each other.
You just have to make sure that the models still fit in the GPU memory.
After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.
The full main function looks like this:
def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
data_dir = os.path.abspath("./data")
load_data(data_dir)
config = {
"l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
scheduler = ASHAScheduler(
metric="loss",
mode="min",
max_t=max_num_epochs,
grace_period=1,
reduction_factor=2)
reporter = CLIReporter(
# parameter_columns=["l1", "l2", "lr", "batch_size"],
metric_columns=["loss", "accuracy", "training_iteration"])
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter)
best_trial = result.get_best_trial("loss", "min", "last")
print("Best trial config: {}".format(best_trial.config))
print("Best trial final validation loss: {}".format(
best_trial.last_result["loss"]))
print("Best trial final validation accuracy: {}".format(
best_trial.last_result["accuracy"]))
best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if gpus_per_trial > 1:
best_trained_model = nn.DataParallel(best_trained_model)
best_trained_model.to(device)
best_checkpoint_dir = best_trial.checkpoint.value
model_state, optimizer_state = torch.load(os.path.join(
best_checkpoint_dir, "checkpoint"))
best_trained_model.load_state_dict(model_state)
test_acc = test_accuracy(best_trained_model, device)
print("Best trial test set accuracy: {}".format(test_acc))
if __name__ == "__main__":
# You can change the number of GPUs per trial here:
main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)
Out:
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz
Extracting /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data
Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:45:29 (running for 00:00:00.20)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | PENDING | | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | PENDING | | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | PENDING | | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
[2m[36m(func pid=5006)[0m Files already downloaded and verified
[2m[36m(func pid=5006)[0m Files already downloaded and verified
[2m[36m(func pid=5009)[0m Files already downloaded and verified
[2m[36m(func pid=5004)[0m Files already downloaded and verified
[2m[36m(func pid=5008)[0m Files already downloaded and verified
[2m[36m(func pid=5009)[0m Files already downloaded and verified
[2m[36m(func pid=5004)[0m Files already downloaded and verified
[2m[36m(func pid=5008)[0m Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:45:34 (running for 00:00:05.25)
Memory usage on this node: 21.4/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
== Status ==
Current time: 2022-04-10 04:45:40 (running for 00:00:11.22)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
[2m[36m(func pid=5004)[0m [1, 2000] loss: 2.161
[2m[36m(func pid=5009)[0m [1, 2000] loss: 2.344
[2m[36m(func pid=5006)[0m [1, 2000] loss: 2.304
[2m[36m(func pid=5008)[0m [1, 2000] loss: 2.280
== Status ==
Current time: 2022-04-10 04:45:45 (running for 00:00:16.24)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
== Status ==
Current time: 2022-04-10 04:45:50 (running for 00:00:21.27)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
[2m[36m(func pid=5004)[0m [1, 4000] loss: 1.003
[2m[36m(func pid=5009)[0m [1, 4000] loss: 1.171
[2m[36m(func pid=5006)[0m [1, 4000] loss: 1.149
== Status ==
Current time: 2022-04-10 04:45:55 (running for 00:00:26.29)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
[2m[36m(func pid=5008)[0m [1, 4000] loss: 1.041
== Status ==
Current time: 2022-04-10 04:46:00 (running for 00:00:31.33)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
[2m[36m(func pid=5004)[0m [1, 6000] loss: 0.647
[2m[36m(func pid=5009)[0m [1, 6000] loss: 0.781
== Status ==
Current time: 2022-04-10 04:46:05 (running for 00:00:36.36)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 |
+---------------------+----------+-----------------+--------------+------+------+-------------+
Result for DEFAULT_0c10e_00000:
accuracy: 0.1328
date: 2022-04-10_04-46-06
done: false
experiment_id: 1f1ce6d2da8349da94a7b7c2a34154b0
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.241459646701813
node_ip: 172.17.0.3
pid: 5006
should_checkpoint: true
time_since_restore: 35.506600856781006
time_this_iter_s: 35.506600856781006
time_total_s: 35.506600856781006
timestamp: 1649565966
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00000
Result for DEFAULT_0c10e_00002:
accuracy: 0.2314
date: 2022-04-10_04-46-06
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 1.9600008015632628
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 34.58101677894592
time_this_iter_s: 34.58101677894592
time_total_s: 34.58101677894592
timestamp: 1649565966
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:46:10 (running for 00:00:41.38)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | | | |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [1, 8000] loss: 0.481
[2m[36m(func pid=5009)[0m [1, 8000] loss: 0.586
== Status ==
Current time: 2022-04-10 04:46:15 (running for 00:00:46.40)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | | | |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5006)[0m [2, 2000] loss: 2.176
[2m[36m(func pid=5008)[0m [2, 2000] loss: 1.881
[2m[36m(func pid=5004)[0m [1, 10000] loss: 0.379
== Status ==
Current time: 2022-04-10 04:46:20 (running for 00:00:51.43)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | | | |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5009)[0m [1, 10000] loss: 0.469
== Status ==
Current time: 2022-04-10 04:46:26 (running for 00:00:56.46)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | | | |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [2, 4000] loss: 0.883
[2m[36m(func pid=5006)[0m [2, 4000] loss: 1.010
[2m[36m(func pid=5004)[0m [1, 12000] loss: 0.318
== Status ==
Current time: 2022-04-10 04:46:31 (running for 00:01:01.49)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00001 | RUNNING | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | | | |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | PENDING | | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00001:
accuracy: 0.1014
date: 2022-04-10_04-46-31
done: true
experiment_id: 03200849f7084895a8375703be778bbf
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.3629231911182402
node_ip: 172.17.0.3
pid: 5009
should_checkpoint: true
time_since_restore: 59.77773141860962
time_this_iter_s: 59.77773141860962
time_total_s: 59.77773141860962
timestamp: 1649565991
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00001
[2m[36m(func pid=5005)[0m Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:46:36 (running for 00:01:07.30)
Memory usage on this node: 21.3/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 2.24146 | 0.1328 | 1 |
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.96 | 0.2314 | 1 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | RUNNING | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5005)[0m Files already downloaded and verified
Result for DEFAULT_0c10e_00002:
accuracy: 0.3859
date: 2022-04-10_04-46-38
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 2
loss: 1.645931950187683
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 66.25421857833862
time_this_iter_s: 31.6732017993927
time_total_s: 66.25421857833862
timestamp: 1649565998
timesteps_since_restore: 0
training_iteration: 2
trial_id: 0c10e_00002
[2m[36m(func pid=5004)[0m [1, 14000] loss: 0.271
Result for DEFAULT_0c10e_00000:
accuracy: 0.2949
date: 2022-04-10_04-46-39
done: true
experiment_id: 1f1ce6d2da8349da94a7b7c2a34154b0
hostname: d52bd7357ae4
iterations_since_restore: 2
loss: 1.9218169279098511
node_ip: 172.17.0.3
pid: 5006
should_checkpoint: true
time_since_restore: 68.59252429008484
time_this_iter_s: 33.08592343330383
time_total_s: 68.59252429008484
timestamp: 1649565999
timesteps_since_restore: 0
training_iteration: 2
trial_id: 0c10e_00000
== Status ==
Current time: 2022-04-10 04:46:42 (running for 00:01:12.85)
Memory usage on this node: 21.2/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (5 PENDING, 3 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | RUNNING | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | PENDING | | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5003)[0m Files already downloaded and verified
[2m[36m(func pid=5003)[0m Files already downloaded and verified
[2m[36m(func pid=5004)[0m [1, 16000] loss: 0.237
== Status ==
Current time: 2022-04-10 04:46:47 (running for 00:01:17.88)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | RUNNING | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [3, 2000] loss: 1.618
[2m[36m(func pid=5005)[0m [1, 2000] loss: 2.302
== Status ==
Current time: 2022-04-10 04:46:52 (running for 00:01:22.93)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | RUNNING | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [1, 18000] loss: 0.211
== Status ==
Current time: 2022-04-10 04:46:57 (running for 00:01:27.95)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00004 | RUNNING | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | PENDING | | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00004:
accuracy: 0.1769
date: 2022-04-10_04-46-58
done: true
experiment_id: 57ed9d0ac381450aa6becf0247e0c785
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.276254405593872
node_ip: 172.17.0.3
pid: 5005
should_checkpoint: true
time_since_restore: 22.50801730155945
time_this_iter_s: 22.50801730155945
time_total_s: 22.50801730155945
timestamp: 1649566018
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00004
[2m[36m(func pid=5003)[0m [1, 2000] loss: 2.225
[2m[36m(func pid=5008)[0m [3, 4000] loss: 0.773
== Status ==
Current time: 2022-04-10 04:47:03 (running for 00:01:33.46)
Memory usage on this node: 20.3/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.2588570261478425
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | | | |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m Files already downloaded and verified
[2m[36m(func pid=5007)[0m Files already downloaded and verified
Result for DEFAULT_0c10e_00005:
accuracy: 0.2201
date: 2022-04-10_04-47-06
done: false
experiment_id: f40073fe155e42f98d126de338b9f00b
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 1.9598983837127686
node_ip: 172.17.0.3
pid: 5003
should_checkpoint: true
time_since_restore: 22.279260635375977
time_this_iter_s: 22.279260635375977
time_total_s: 22.279260635375977
timestamp: 1649566026
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00005
[2m[36m(func pid=5004)[0m [1, 20000] loss: 0.188
== Status ==
Current time: 2022-04-10 04:47:08 (running for 00:01:38.71)
Memory usage on this node: 20.6/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.64593 | 0.3859 | 2 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.9599 | 0.2201 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00002:
accuracy: 0.4493
date: 2022-04-10_04-47-11
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 3
loss: 1.498528915500641
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 99.243403673172
time_this_iter_s: 32.989185094833374
time_total_s: 99.243403673172
timestamp: 1649566031
timesteps_since_restore: 0
training_iteration: 3
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:47:13 (running for 00:01:44.01)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.9599 | 0.2201 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [1, 2000] loss: 2.200
== Status ==
Current time: 2022-04-10 04:47:18 (running for 00:01:49.04)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | | | |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.9599 | 0.2201 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5003)[0m [2, 2000] loss: 1.809
[2m[36m(func pid=5008)[0m [4, 2000] loss: 1.480
Result for DEFAULT_0c10e_00003:
accuracy: 0.2992
date: 2022-04-10_04-47-23
done: false
experiment_id: 3c43cf11e6bf473a9355f90923040cdd
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 1.8393557698599994
node_ip: 172.17.0.3
pid: 5004
should_checkpoint: true
time_since_restore: 111.17262768745422
time_this_iter_s: 111.17262768745422
time_total_s: 111.17262768745422
timestamp: 1649566043
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00003
== Status ==
Current time: 2022-04-10 04:47:24 (running for 00:01:54.73)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.9599 | 0.2201 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [1, 4000] loss: 1.015
Result for DEFAULT_0c10e_00005:
accuracy: 0.3477
date: 2022-04-10_04-47-26
done: false
experiment_id: f40073fe155e42f98d126de338b9f00b
hostname: d52bd7357ae4
iterations_since_restore: 2
loss: 1.6899889686584473
node_ip: 172.17.0.3
pid: 5003
should_checkpoint: true
time_since_restore: 42.91800880432129
time_this_iter_s: 20.638748168945312
time_total_s: 42.91800880432129
timestamp: 1649566046
timesteps_since_restore: 0
training_iteration: 2
trial_id: 0c10e_00005
== Status ==
Current time: 2022-04-10 04:47:29 (running for 00:02:00.35)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.68999 | 0.3477 | 2 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [2, 2000] loss: 1.893
[2m[36m(func pid=5008)[0m [4, 4000] loss: 0.723
[2m[36m(func pid=5007)[0m [1, 6000] loss: 0.637
== Status ==
Current time: 2022-04-10 04:47:34 (running for 00:02:05.39)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.68999 | 0.3477 | 2 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
== Status ==
Current time: 2022-04-10 04:47:39 (running for 00:02:10.42)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.49853 | 0.4493 | 3 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.68999 | 0.3477 | 2 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5003)[0m [3, 2000] loss: 1.663
[2m[36m(func pid=5004)[0m [2, 4000] loss: 0.955
[2m[36m(func pid=5007)[0m [1, 8000] loss: 0.463
Result for DEFAULT_0c10e_00002:
accuracy: 0.5001
date: 2022-04-10_04-47-44
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 4
loss: 1.3877728350639342
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 132.1592869758606
time_this_iter_s: 32.9158833026886
time_total_s: 132.1592869758606
timestamp: 1649566064
timesteps_since_restore: 0
training_iteration: 4
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:47:45 (running for 00:02:15.94)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.68999 | 0.3477 | 2 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00005:
accuracy: 0.3869
date: 2022-04-10_04-47-47
done: false
experiment_id: f40073fe155e42f98d126de338b9f00b
hostname: d52bd7357ae4
iterations_since_restore: 3
loss: 1.601786145591736
node_ip: 172.17.0.3
pid: 5003
should_checkpoint: true
time_since_restore: 63.52823877334595
time_this_iter_s: 20.610229969024658
time_total_s: 63.52823877334595
timestamp: 1649566067
timesteps_since_restore: 0
training_iteration: 3
trial_id: 0c10e_00005
== Status ==
Current time: 2022-04-10 04:47:50 (running for 00:02:20.96)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.60179 | 0.3869 | 3 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [2, 6000] loss: 0.635
[2m[36m(func pid=5007)[0m [1, 10000] loss: 0.358
== Status ==
Current time: 2022-04-10 04:47:55 (running for 00:02:25.99)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.60179 | 0.3869 | 3 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [5, 2000] loss: 1.392
== Status ==
Current time: 2022-04-10 04:48:00 (running for 00:02:31.02)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.60179 | 0.3869 | 3 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [2, 8000] loss: 0.476
[2m[36m(func pid=5003)[0m [4, 2000] loss: 1.581
[2m[36m(func pid=5007)[0m [1, 12000] loss: 0.293
== Status ==
Current time: 2022-04-10 04:48:05 (running for 00:02:36.06)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00005 | RUNNING | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.60179 | 0.3869 | 3 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [5, 4000] loss: 0.680
Result for DEFAULT_0c10e_00005:
accuracy: 0.393
date: 2022-04-10_04-48-08
done: true
experiment_id: f40073fe155e42f98d126de338b9f00b
hostname: d52bd7357ae4
iterations_since_restore: 4
loss: 1.5446142356872559
node_ip: 172.17.0.3
pid: 5003
should_checkpoint: true
time_since_restore: 84.13721871376038
time_this_iter_s: 20.60897994041443
time_total_s: 84.13721871376038
timestamp: 1649566088
timesteps_since_restore: 0
training_iteration: 4
trial_id: 0c10e_00005
[2m[36m(func pid=5004)[0m [2, 10000] loss: 0.380
== Status ==
Current time: 2022-04-10 04:48:11 (running for 00:02:41.57)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 3 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | PENDING | | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [1, 14000] loss: 0.251
[2m[36m(func pid=5002)[0m Files already downloaded and verified
[2m[36m(func pid=5002)[0m Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:48:16 (running for 00:02:46.62)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.38777 | 0.5001 | 4 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | RUNNING | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00002:
accuracy: 0.5279
date: 2022-04-10_04-48-17
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 5
loss: 1.3209753231763839
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 165.1197385787964
time_this_iter_s: 32.96045160293579
time_total_s: 165.1197385787964
timestamp: 1649566097
timesteps_since_restore: 0
training_iteration: 5
trial_id: 0c10e_00002
[2m[36m(func pid=5004)[0m [2, 12000] loss: 0.325
[2m[36m(func pid=5007)[0m [1, 16000] loss: 0.219
== Status ==
Current time: 2022-04-10 04:48:21 (running for 00:02:51.89)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | RUNNING | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
== Status ==
Current time: 2022-04-10 04:48:26 (running for 00:02:56.93)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | RUNNING | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5002)[0m [1, 2000] loss: 2.294
[2m[36m(func pid=5004)[0m [2, 14000] loss: 0.278
[2m[36m(func pid=5008)[0m [6, 2000] loss: 1.306
[2m[36m(func pid=5007)[0m [1, 18000] loss: 0.200
== Status ==
Current time: 2022-04-10 04:48:31 (running for 00:03:01.96)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00007 | RUNNING | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00007:
accuracy: 0.1021
date: 2022-04-10_04-48-35
done: true
experiment_id: 8c02aed1ca0b430f8522294784a6b331
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.3056997585296632
node_ip: 172.17.0.3
pid: 5002
should_checkpoint: true
time_since_restore: 22.65566062927246
time_this_iter_s: 22.65566062927246
time_total_s: 22.65566062927246
timestamp: 1649566115
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00007
== Status ==
Current time: 2022-04-10 04:48:37 (running for 00:03:07.75)
Memory usage on this node: 20.3/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 3 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00008 | PENDING | | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5004)[0m [2, 16000] loss: 0.239
[2m[36m(func pid=5007)[0m [1, 20000] loss: 0.176
[2m[36m(func pid=5008)[0m [6, 4000] loss: 0.651
[2m[36m(func pid=8344)[0m Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:48:42 (running for 00:03:12.78)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00008 | RUNNING | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=8344)[0m Files already downloaded and verified
[2m[36m(func pid=5004)[0m [2, 18000] loss: 0.215
== Status ==
Current time: 2022-04-10 04:48:47 (running for 00:03:17.83)
Memory usage on this node: 20.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.32098 | 0.5279 | 5 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00008 | RUNNING | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00002:
accuracy: 0.5379
date: 2022-04-10_04-48-50
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 6
loss: 1.279394944357872
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 197.71211290359497
time_this_iter_s: 32.592374324798584
time_total_s: 197.71211290359497
timestamp: 1649566130
timesteps_since_restore: 0
training_iteration: 6
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:48:53 (running for 00:03:23.49)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | | | |
| DEFAULT_0c10e_00008 | RUNNING | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00006:
accuracy: 0.3431
date: 2022-04-10_04-48-55
done: false
experiment_id: 59808294236046a390077b9dc120b8cf
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 1.7145750304013492
node_ip: 172.17.0.3
pid: 5007
should_checkpoint: true
time_since_restore: 111.48537373542786
time_this_iter_s: 111.48537373542786
time_total_s: 111.48537373542786
timestamp: 1649566135
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00006
[2m[36m(func pid=5004)[0m [2, 20000] loss: 0.197
[2m[36m(func pid=8344)[0m [1, 2000] loss: 2.313
== Status ==
Current time: 2022-04-10 04:48:58 (running for 00:03:28.59)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00008 | RUNNING | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | | | |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [7, 2000] loss: 1.246
Result for DEFAULT_0c10e_00008:
accuracy: 0.1007
date: 2022-04-10_04-49-02
done: true
experiment_id: 1b12f08297d74f769c4bf4aca1a85bce
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.3078824977874755
node_ip: 172.17.0.3
pid: 8344
should_checkpoint: true
time_since_restore: 22.114116430282593
time_this_iter_s: 22.114116430282593
time_total_s: 22.114116430282593
timestamp: 1649566142
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00008
== Status ==
Current time: 2022-04-10 04:49:04 (running for 00:03:34.46)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 3 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | PENDING | | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 2000] loss: 1.713
[2m[36m(func pid=9009)[0m Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:49:09 (running for 00:03:39.50)
Memory usage on this node: 20.4/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00003 | RUNNING | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 1.83936 | 0.2992 | 1 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=9009)[0m Files already downloaded and verified
Result for DEFAULT_0c10e_00003:
accuracy: 0.2373
date: 2022-04-10_04-49-10
done: true
experiment_id: 3c43cf11e6bf473a9355f90923040cdd
hostname: d52bd7357ae4
iterations_since_restore: 2
loss: 2.0048313949137926
node_ip: 172.17.0.3
pid: 5004
should_checkpoint: true
time_since_restore: 218.33158326148987
time_this_iter_s: 107.15895557403564
time_total_s: 218.33158326148987
timestamp: 1649566150
timesteps_since_restore: 0
training_iteration: 2
trial_id: 0c10e_00003
[2m[36m(func pid=5008)[0m [7, 4000] loss: 0.628
[2m[36m(func pid=5007)[0m [2, 4000] loss: 0.865
== Status ==
Current time: 2022-04-10 04:49:14 (running for 00:03:44.88)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 6000] loss: 0.576
== Status ==
Current time: 2022-04-10 04:49:19 (running for 00:03:49.90)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.27939 | 0.5379 | 6 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00002:
accuracy: 0.5528
date: 2022-04-10_04-49-19
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 7
loss: 1.2448112061977386
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 227.53409790992737
time_this_iter_s: 29.821985006332397
time_total_s: 227.53409790992737
timestamp: 1649566159
timesteps_since_restore: 0
training_iteration: 7
trial_id: 0c10e_00002
[2m[36m(func pid=9009)[0m [1, 2000] loss: 2.344
== Status ==
Current time: 2022-04-10 04:49:24 (running for 00:03:55.30)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.24481 | 0.5528 | 7 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 8000] loss: 0.428
[2m[36m(func pid=5008)[0m [8, 2000] loss: 1.209
== Status ==
Current time: 2022-04-10 04:49:29 (running for 00:04:00.31)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.24481 | 0.5528 | 7 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=9009)[0m [1, 4000] loss: 1.168
[2m[36m(func pid=5007)[0m [2, 10000] loss: 0.347
== Status ==
Current time: 2022-04-10 04:49:34 (running for 00:04:05.33)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.24481 | 0.5528 | 7 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [8, 4000] loss: 0.603
[2m[36m(func pid=5007)[0m [2, 12000] loss: 0.291
== Status ==
Current time: 2022-04-10 04:49:39 (running for 00:04:10.34)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.24481 | 0.5528 | 7 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=9009)[0m [1, 6000] loss: 0.780
== Status ==
Current time: 2022-04-10 04:49:44 (running for 00:04:15.35)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.24481 | 0.5528 | 7 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00002:
accuracy: 0.5533
date: 2022-04-10_04-49-46
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 8
loss: 1.2396061965227128
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 253.7500820159912
time_this_iter_s: 26.215984106063843
time_total_s: 253.7500820159912
timestamp: 1649566186
timesteps_since_restore: 0
training_iteration: 8
trial_id: 0c10e_00002
[2m[36m(func pid=5007)[0m [2, 14000] loss: 0.249
== Status ==
Current time: 2022-04-10 04:49:50 (running for 00:04:20.52)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.23961 | 0.5533 | 8 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 16000] loss: 0.220
[2m[36m(func pid=9009)[0m [1, 8000] loss: 0.585
[2m[36m(func pid=5008)[0m [9, 2000] loss: 1.171
== Status ==
Current time: 2022-04-10 04:49:55 (running for 00:04:25.53)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.23961 | 0.5533 | 8 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 18000] loss: 0.193
== Status ==
Current time: 2022-04-10 04:50:00 (running for 00:04:30.55)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.23961 | 0.5533 | 8 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [9, 4000] loss: 0.588
[2m[36m(func pid=9009)[0m [1, 10000] loss: 0.468
== Status ==
Current time: 2022-04-10 04:50:05 (running for 00:04:35.56)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.23961 | 0.5533 | 8 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [2, 20000] loss: 0.174
== Status ==
Current time: 2022-04-10 04:50:10 (running for 00:04:40.57)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.23961 | 0.5533 | 8 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00009 | RUNNING | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | | | |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00009:
accuracy: 0.1029
date: 2022-04-10_04-50-11
done: true
experiment_id: 8bd00d26b5494a6eb8a6e14f7f4cd522
hostname: d52bd7357ae4
iterations_since_restore: 1
loss: 2.325470637321472
node_ip: 172.17.0.3
pid: 9009
should_checkpoint: true
time_since_restore: 63.21910643577576
time_this_iter_s: 63.21910643577576
time_total_s: 63.21910643577576
timestamp: 1649566211
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0c10e_00009
Result for DEFAULT_0c10e_00002:
accuracy: 0.5576
date: 2022-04-10_04-50-12
done: false
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 9
loss: 1.2259806815862655
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 279.8658182621002
time_this_iter_s: 26.11573624610901
time_total_s: 279.8658182621002
timestamp: 1649566212
timesteps_since_restore: 0
training_iteration: 9
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:50:15 (running for 00:04:45.63)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22598 | 0.5576 | 9 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.71458 | 0.3431 | 1 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00006:
accuracy: 0.3272
date: 2022-04-10_04-50-17
done: false
experiment_id: 59808294236046a390077b9dc120b8cf
hostname: d52bd7357ae4
iterations_since_restore: 2
loss: 1.782580977988243
node_ip: 172.17.0.3
pid: 5007
should_checkpoint: true
time_since_restore: 193.41082763671875
time_this_iter_s: 81.9254539012909
time_total_s: 193.41082763671875
timestamp: 1649566217
timesteps_since_restore: 0
training_iteration: 2
trial_id: 0c10e_00006
[2m[36m(func pid=5008)[0m [10, 2000] loss: 1.144
== Status ==
Current time: 2022-04-10 04:50:21 (running for 00:04:51.51)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22598 | 0.5576 | 9 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 2000] loss: 1.718
== Status ==
Current time: 2022-04-10 04:50:26 (running for 00:04:56.52)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22598 | 0.5576 | 9 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5008)[0m [10, 4000] loss: 0.579
[2m[36m(func pid=5007)[0m [3, 4000] loss: 0.853
== Status ==
Current time: 2022-04-10 04:50:31 (running for 00:05:01.53)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22598 | 0.5576 | 9 |
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 6000] loss: 0.583
Result for DEFAULT_0c10e_00002:
accuracy: 0.5627
date: 2022-04-10_04-50-35
done: true
experiment_id: fd74a8193f3640b1be44dc88dd569b6a
hostname: d52bd7357ae4
iterations_since_restore: 10
loss: 1.222454554271698
node_ip: 172.17.0.3
pid: 5008
should_checkpoint: true
time_since_restore: 303.0206735134125
time_this_iter_s: 23.154855251312256
time_total_s: 303.0206735134125
timestamp: 1649566235
timesteps_since_restore: 0
training_iteration: 10
trial_id: 0c10e_00002
== Status ==
Current time: 2022-04-10 04:50:36 (running for 00:05:06.78)
Memory usage on this node: 19.3/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 8000] loss: 0.435
== Status ==
Current time: 2022-04-10 04:50:41 (running for 00:05:11.80)
Memory usage on this node: 19.3/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 10000] loss: 0.351
== Status ==
Current time: 2022-04-10 04:50:46 (running for 00:05:16.81)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 12000] loss: 0.285
== Status ==
Current time: 2022-04-10 04:50:51 (running for 00:05:21.82)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 14000] loss: 0.250
== Status ==
Current time: 2022-04-10 04:50:56 (running for 00:05:26.82)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 16000] loss: 0.216
== Status ==
Current time: 2022-04-10 04:51:01 (running for 00:05:31.84)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 18000] loss: 0.196
== Status ==
Current time: 2022-04-10 04:51:06 (running for 00:05:36.84)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [3, 20000] loss: 0.170
== Status ==
Current time: 2022-04-10 04:51:11 (running for 00:05:41.86)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.78258 | 0.3272 | 2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00006:
accuracy: 0.3594
date: 2022-04-10_04-51-15
done: false
experiment_id: 59808294236046a390077b9dc120b8cf
hostname: d52bd7357ae4
iterations_since_restore: 3
loss: 1.7754648028180002
node_ip: 172.17.0.3
pid: 5007
should_checkpoint: true
time_since_restore: 251.42181205749512
time_this_iter_s: 58.01098442077637
time_total_s: 251.42181205749512
timestamp: 1649566275
timesteps_since_restore: 0
training_iteration: 3
trial_id: 0c10e_00006
== Status ==
Current time: 2022-04-10 04:51:17 (running for 00:05:47.50)
Memory usage on this node: 19.6/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 2000] loss: 1.746
== Status ==
Current time: 2022-04-10 04:51:22 (running for 00:05:52.52)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 4000] loss: 0.861
== Status ==
Current time: 2022-04-10 04:51:27 (running for 00:05:57.52)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 6000] loss: 0.560
== Status ==
Current time: 2022-04-10 04:51:32 (running for 00:06:02.54)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 8000] loss: 0.445
== Status ==
Current time: 2022-04-10 04:51:37 (running for 00:06:07.54)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 10000] loss: 0.349
== Status ==
Current time: 2022-04-10 04:51:42 (running for 00:06:12.56)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 12000] loss: 0.292
== Status ==
Current time: 2022-04-10 04:51:47 (running for 00:06:17.57)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 14000] loss: 0.250
== Status ==
Current time: 2022-04-10 04:51:52 (running for 00:06:22.58)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 16000] loss: 0.217
[2m[36m(func pid=5007)[0m [4, 18000] loss: 0.193
== Status ==
Current time: 2022-04-10 04:51:57 (running for 00:06:27.59)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(func pid=5007)[0m [4, 20000] loss: 0.171
== Status ==
Current time: 2022-04-10 04:52:02 (running for 00:06:32.60)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
== Status ==
Current time: 2022-04-10 04:52:07 (running for 00:06:37.61)
Memory usage on this node: 20.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.77546 | 0.3594 | 3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_0c10e_00006:
accuracy: 0.3782
date: 2022-04-10_04-52-09
done: true
experiment_id: 59808294236046a390077b9dc120b8cf
hostname: d52bd7357ae4
iterations_since_restore: 4
loss: 1.7262782471820712
node_ip: 172.17.0.3
pid: 5007
should_checkpoint: true
time_since_restore: 305.57490062713623
time_this_iter_s: 54.15308856964111
time_total_s: 305.57490062713623
timestamp: 1649566329
timesteps_since_restore: 0
training_iteration: 4
trial_id: 0c10e_00006
== Status ==
Current time: 2022-04-10 04:52:09 (running for 00:06:39.66)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.5446142356872559 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 | 8 | 128 | 32 | 0.000234003 | 1.92182 | 0.2949 | 2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 | 4 | 4 | 4 | 0.073943 | 2.36292 | 0.1014 | 1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 | 8 | 32 | 8 | 0.000505412 | 1.22245 | 0.5627 | 10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 | 2 | 16 | 16 | 0.00365822 | 2.00483 | 0.2373 | 2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 | 16 | 4 | 128 | 0.000156333 | 2.27625 | 0.1769 | 1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 | 16 | 8 | 4 | 0.00147991 | 1.54461 | 0.393 | 4 |
| DEFAULT_0c10e_00006 | TERMINATED | 172.17.0.3:5007 | 2 | 8 | 8 | 0.00251674 | 1.72628 | 0.3782 | 4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 | 16 | 8 | 16 | 0.0569105 | 2.3057 | 0.1021 | 1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 | 16 | 16 | 16 | 0.0642732 | 2.30788 | 0.1007 | 1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 | 4 | 256 | 64 | 0.0648686 | 2.32547 | 0.1029 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Best trial config: {'l1': 32, 'l2': 8, 'lr': 0.0005054115351056523, 'batch_size': 8}
Best trial final validation loss: 1.222454554271698
Best trial final validation accuracy: 0.5627
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.5672
If you run the code, an example output could look like this:
Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... | l1 | l2 | lr | batch_size | loss | accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... | 64 | 4 | 0.00011629 | 2 | 1.87273 | 0.244 | 2 |
| ... | 32 | 64 | 0.000339763 | 8 | 1.23603 | 0.567 | 8 |
| ... | 8 | 16 | 0.00276249 | 16 | 1.1815 | 0.5836 | 10 |
| ... | 4 | 64 | 0.000648721 | 4 | 1.31131 | 0.5224 | 8 |
| ... | 32 | 16 | 0.000340753 | 8 | 1.26454 | 0.5444 | 8 |
| ... | 8 | 4 | 0.000699775 | 8 | 1.99594 | 0.1983 | 2 |
| ... | 256 | 8 | 0.0839654 | 16 | 2.3119 | 0.0993 | 1 |
| ... | 16 | 128 | 0.0758154 | 16 | 2.33575 | 0.1327 | 1 |
| ... | 16 | 8 | 0.0763312 | 16 | 2.31129 | 0.1042 | 4 |
| ... | 128 | 16 | 0.000124903 | 4 | 2.26917 | 0.1945 | 1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+
Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806
Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.
So that’s it! You can now tune the parameters of your PyTorch models.
Total running time of the script: ( 7 minutes 13.842 seconds)