Hyperparameter tuning with Ray Tune

Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.

Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.

In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.

As you will see, we only need to add some slight modifications. In particular, we need to

1. wrap data loading and training in functions,
2. make some network parameters configurable,
3. add checkpointing (optional),
4. and define the search space for the model tuning

To run this tutorial, please make sure the following packages are installed:

• ray[tune]: Distributed hyperparameter tuning library
• torchvision: For the data transformers

Setup / Imports

Let’s start with the imports:

from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler

Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.

Data loaders

We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.

def load_data(data_dir="./data"):
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    trainset = torchvision.datasets.CIFAR10(
        root=data_dir, train=True, download=True, transform=transform)

    testset = torchvision.datasets.CIFAR10(
        root=data_dir, train=False, download=True, transform=transform)

    return trainset, testset

Configurable neural network

We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The train function

Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.

We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None). As you can guess, the config parameter will receive the hyperparameters we would like to train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies the directory where we load and store the data, so multiple runs can share the same data source.

net = Net(config["l1"], config["l2"])

if checkpoint_dir:
    model_state, optimizer_state = torch.load(
        os.path.join(checkpoint_dir, "checkpoint"))
    net.load_state_dict(model_state)
    optimizer.load_state_dict(optimizer_state)

The learning rate of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.

Adding (multi) GPU support with DataParallel

Image classification benefits largely from GPUs. Luckily, we can continue to use PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel to support data parallel training on multiple GPUs:

device = "cpu"
if torch.cuda.is_available():
    device = "cuda:0"
    if torch.cuda.device_count() > 1:
        net = nn.DataParallel(net)
net.to(device)

By using a device variable we make sure that training also works when we have no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly, like this:

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.

Communicating with Ray Tune

The most interesting part is the communication with Ray Tune:

with tune.checkpoint_dir(epoch) as checkpoint_dir:
    path = os.path.join(checkpoint_dir, "checkpoint")
    torch.save((net.state_dict(), optimizer.state_dict()), path)

tune.report(loss=(val_loss / val_steps), accuracy=correct / total)

Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.

The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.

Full training function

The full code example looks like this:

def train_cifar(config, checkpoint_dir=None, data_dir=None):
    net = Net(config["l1"], config["l2"])

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        net.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    trainset, testset = load_data(data_dir)

    test_abs = int(len(trainset) * 0.8)
    train_subset, val_subset = random_split(
        trainset, [test_abs, len(trainset) - test_abs])

    trainloader = torch.utils.data.DataLoader(
        train_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)
    valloader = torch.utils.data.DataLoader(
        val_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)

    for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        epoch_steps = 0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            epoch_steps += 1
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
                                                running_loss / epoch_steps))
                running_loss = 0.0

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0
        for i, data in enumerate(valloader, 0):
            with torch.no_grad():
                inputs, labels = data
                inputs, labels = inputs.to(device), labels.to(device)

                outputs = net(inputs)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

                loss = criterion(outputs, labels)
                val_loss += loss.cpu().numpy()
                val_steps += 1

        with tune.checkpoint_dir(epoch) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((net.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
    print("Finished Training")

As you can see, most of the code is adapted directly from the original example.

Test set accuracy

Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:

def test_accuracy(net, device="cpu"):
    trainset, testset = load_data()

    testloader = torch.utils.data.DataLoader(
        testset, batch_size=4, shuffle=False, num_workers=2)

    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

The function also expects a device parameter, so we can do the test set validation on a GPU.

Configuring the search space

Lastly, we need to define Ray Tune’s search space. Here is an example:

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function makes it possible to define your own sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, Ray Tune will now randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. We also use the ASHAScheduler which will terminate bad performing trials early.

We wrap the train_cifar function with functools.partial to set the constant data_dir parameter. We can also tell Ray Tune what resources should be available for each trial:

gpus_per_trial = 2
# ...
result = tune.run(
    partial(train_cifar, data_dir=data_dir),
    resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
    config=config,
    num_samples=num_samples,
    scheduler=scheduler,
    progress_reporter=reporter,
    checkpoint_at_end=True)

You can specify the number of CPUs, which are then available e.g. to increase the num_workers of the PyTorch DataLoader instances. The selected number of GPUs are made visible to PyTorch in each trial. Trials do not have access to GPUs that haven’t been requested for them - so you don’t have to care about two trials using the same set of resources.

Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is completely valid. The trials will then share GPUs among each other. You just have to make sure that the models still fit in the GPU memory.

After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.

The full main function looks like this:

def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
    data_dir = os.path.abspath("./data")
    load_data(data_dir)
    config = {
        "l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }
    scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=max_num_epochs,
        grace_period=1,
        reduction_factor=2)
    reporter = CLIReporter(
        # parameter_columns=["l1", "l2", "lr", "batch_size"],
        metric_columns=["loss", "accuracy", "training_iteration"])
    result = tune.run(
        partial(train_cifar, data_dir=data_dir),
        resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
        config=config,
        num_samples=num_samples,
        scheduler=scheduler,
        progress_reporter=reporter)

    best_trial = result.get_best_trial("loss", "min", "last")
    print("Best trial config: {}".format(best_trial.config))
    print("Best trial final validation loss: {}".format(
        best_trial.last_result["loss"]))
    print("Best trial final validation accuracy: {}".format(
        best_trial.last_result["accuracy"]))

    best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if gpus_per_trial > 1:
            best_trained_model = nn.DataParallel(best_trained_model)
    best_trained_model.to(device)

    best_checkpoint_dir = best_trial.checkpoint.value
    model_state, optimizer_state = torch.load(os.path.join(
        best_checkpoint_dir, "checkpoint"))
    best_trained_model.load_state_dict(model_state)

    test_acc = test_accuracy(best_trained_model, device)
    print("Best trial test set accuracy: {}".format(test_acc))


if __name__ == "__main__":
    # You can change the number of GPUs per trial here:
    main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)

Out:

Files already downloaded and verified
Files already downloaded and verified
== Status ==
Memory usage on this node: 6.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_61189_00000 | RUNNING  |       |            2 |   32 |    4 | 0.00184559  |
| DEFAULT_61189_00001 | PENDING  |       |            4 |   16 |  256 | 0.055604    |
| DEFAULT_61189_00002 | PENDING  |       |           16 |   32 |   64 | 0.00028312  |
| DEFAULT_61189_00003 | PENDING  |       |            8 |  128 |  128 | 0.000251858 |
| DEFAULT_61189_00004 | PENDING  |       |            4 |    8 |   64 | 0.0112633   |
| DEFAULT_61189_00005 | PENDING  |       |            2 |    8 |  256 | 0.000214909 |
| DEFAULT_61189_00006 | PENDING  |       |            4 |   64 |   32 | 0.000998733 |
| DEFAULT_61189_00007 | PENDING  |       |            8 |   64 |   64 | 0.00120993  |
| DEFAULT_61189_00008 | PENDING  |       |            4 |    4 |  128 | 0.00764251  |
| DEFAULT_61189_00009 | PENDING  |       |           16 |    8 |  256 | 0.00387042  |
+---------------------+----------+-------+--------------+------+------+-------------+


(pid=4799) Files already downloaded and verified
(pid=4804) Files already downloaded and verified
(pid=4805) Files already downloaded and verified
(pid=4806) Files already downloaded and verified
(pid=4804) Files already downloaded and verified
(pid=4799) Files already downloaded and verified
(pid=4805) Files already downloaded and verified
(pid=4806) Files already downloaded and verified
== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_61189_00000 | RUNNING  |       |            2 |   32 |    4 | 0.00184559  |
| DEFAULT_61189_00001 | RUNNING  |       |            4 |   16 |  256 | 0.055604    |
| DEFAULT_61189_00002 | RUNNING  |       |           16 |   32 |   64 | 0.00028312  |
| DEFAULT_61189_00003 | RUNNING  |       |            8 |  128 |  128 | 0.000251858 |
| DEFAULT_61189_00004 | PENDING  |       |            4 |    8 |   64 | 0.0112633   |
| DEFAULT_61189_00005 | PENDING  |       |            2 |    8 |  256 | 0.000214909 |
| DEFAULT_61189_00006 | PENDING  |       |            4 |   64 |   32 | 0.000998733 |
| DEFAULT_61189_00007 | PENDING  |       |            8 |   64 |   64 | 0.00120993  |
| DEFAULT_61189_00008 | PENDING  |       |            4 |    4 |  128 | 0.00764251  |
| DEFAULT_61189_00009 | PENDING  |       |           16 |    8 |  256 | 0.00387042  |
+---------------------+----------+-------+--------------+------+------+-------------+


(pid=4799) [1,  2000] loss: 2.229
(pid=4805) [1,  2000] loss: 2.303
(pid=4806) [1,  2000] loss: 2.337
(pid=4804) [1,  2000] loss: 2.300
Result for DEFAULT_61189_00002:
  accuracy: 0.1463
  date: 2021-05-17_12-22-07
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 2.272225736999512
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 48.20549535751343
  time_this_iter_s: 48.20549535751343
  time_total_s: 48.20549535751343
  timestamp: 1621254127
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING  |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00001 | RUNNING  |                 |            4 |   16 |  256 | 0.055604    |         |            |                      |
| DEFAULT_61189_00002 | RUNNING  | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 2.27223 |     0.1463 |                    1 |
| DEFAULT_61189_00003 | RUNNING  |                 |            8 |  128 |  128 | 0.000251858 |         |            |                      |
| DEFAULT_61189_00004 | PENDING  |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING  |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING  |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING  |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING  |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING  |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [1,  4000] loss: 0.978
(pid=4805) [1,  4000] loss: 1.147
(pid=4806) [1,  4000] loss: 1.166
(pid=4799) [1,  6000] loss: 0.621
(pid=4804) [2,  2000] loss: 2.187
(pid=4806) [1,  6000] loss: 0.778
Result for DEFAULT_61189_00003:
  accuracy: 0.2177
  date: 2021-05-17_12-22-42
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 2.1926736712455748
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 83.35883784294128
  time_this_iter_s: 83.35883784294128
  time_total_s: 83.35883784294128
  timestamp: 1621254162
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.2324497041225433
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING  |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00001 | RUNNING  |                 |            4 |   16 |  256 | 0.055604    |         |            |                      |
| DEFAULT_61189_00002 | RUNNING  | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 2.27223 |     0.1463 |                    1 |
| DEFAULT_61189_00003 | RUNNING  | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 2.19267 |     0.2177 |                    1 |
| DEFAULT_61189_00004 | PENDING  |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING  |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING  |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING  |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING  |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING  |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00002:
  accuracy: 0.2811
  date: 2021-05-17_12-22-50
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 2.012425883293152
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 91.22370553016663
  time_this_iter_s: 43.0182101726532
  time_total_s: 91.22370553016663
  timestamp: 1621254170
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.012425883293152 | Iter 1.000: -2.2324497041225433
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING  |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00001 | RUNNING  |                 |            4 |   16 |  256 | 0.055604    |         |            |                      |
| DEFAULT_61189_00002 | RUNNING  | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 2.01243 |     0.2811 |                    2 |
| DEFAULT_61189_00003 | RUNNING  | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 2.19267 |     0.2177 |                    1 |
| DEFAULT_61189_00004 | PENDING  |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING  |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING  |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING  |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING  |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING  |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [1,  8000] loss: 0.444
(pid=4806) [1,  8000] loss: 0.583
(pid=4805) [2,  2000] loss: 2.056
(pid=4799) [1, 10000] loss: 0.351
(pid=4804) [3,  2000] loss: 1.950
(pid=4806) [1, 10000] loss: 0.467
Result for DEFAULT_61189_00002:
  accuracy: 0.3143
  date: 2021-05-17_12-23-36
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.8553591501235962
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 136.97479367256165
  time_this_iter_s: 45.75108814239502
  time_total_s: 136.97479367256165
  timestamp: 1621254216
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.012425883293152 | Iter 1.000: -2.2324497041225433
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING  |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00001 | RUNNING  |                 |            4 |   16 |  256 | 0.055604    |         |            |                      |
| DEFAULT_61189_00002 | RUNNING  | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.85536 |     0.3143 |                    3 |
| DEFAULT_61189_00003 | RUNNING  | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 2.19267 |     0.2177 |                    1 |
| DEFAULT_61189_00004 | PENDING  |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING  |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING  |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING  |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING  |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING  |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [1, 12000] loss: 0.288
(pid=4805) [2,  4000] loss: 0.955
Result for DEFAULT_61189_00001:
  accuracy: 0.1026
  date: 2021-05-17_12-23-57
  done: true
  experiment_id: cdc3227c8bdc4df78144fe8e9079b967
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 2.3298936381816864
  node_ip: 172.17.0.4
  pid: 4806
  should_checkpoint: true
  time_since_restore: 158.16219019889832
  time_this_iter_s: 158.16219019889832
  time_total_s: 158.16219019889832
  timestamp: 1621254237
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00001'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.012425883293152 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING  |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00001 | RUNNING  | 172.17.0.4:4806 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | RUNNING  | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.85536 |     0.3143 |                    3 |
| DEFAULT_61189_00003 | RUNNING  | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 2.19267 |     0.2177 |                    1 |
| DEFAULT_61189_00004 | PENDING  |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING  |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING  |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING  |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING  |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING  |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) Files already downloaded and verified
(pid=4800) Files already downloaded and verified
Result for DEFAULT_61189_00003:
  accuracy: 0.3465
  date: 2021-05-17_12-24-04
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.7798762449741363
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 164.81846952438354
  time_this_iter_s: 81.45963168144226
  time_total_s: 164.81846952438354
  timestamp: 1621254244
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.85536 |     0.3143 |                    3 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.77988 |     0.3465 |                    2 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [1, 14000] loss: 0.244
(pid=4804) [4,  2000] loss: 1.800
Result for DEFAULT_61189_00002:
  accuracy: 0.3609
  date: 2021-05-17_12-24-20
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.7205610649108887
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 180.91141200065613
  time_this_iter_s: 43.93661832809448
  time_total_s: 180.91141200065613
  timestamp: 1621254260
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.72056 |     0.3609 |                    4 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.77988 |     0.3465 |                    2 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) [1,  2000] loss: 2.141
(pid=4799) [1, 16000] loss: 0.213
(pid=4805) [3,  2000] loss: 1.749
(pid=4800) [1,  4000] loss: 1.030
(pid=4804) [5,  2000] loss: 1.688
(pid=4799) [1, 18000] loss: 0.184
(pid=4805) [3,  4000] loss: 0.823
Result for DEFAULT_61189_00002:
  accuracy: 0.392
  date: 2021-05-17_12-25-07
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.6389908283233643
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 228.07032704353333
  time_this_iter_s: 47.1589150428772
  time_total_s: 228.07032704353333
  timestamp: 1621254307
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.63899 |     0.392  |                    5 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.77988 |     0.3465 |                    2 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) [1,  6000] loss: 0.692
(pid=4799) [1, 20000] loss: 0.168
Result for DEFAULT_61189_00003:
  accuracy: 0.416
  date: 2021-05-17_12-25-25
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.5751190453529358
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 246.52582693099976
  time_this_iter_s: 81.70735740661621
  time_total_s: 246.52582693099976
  timestamp: 1621254325
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.63899 |     0.392  |                    5 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.57512 |     0.416  |                    3 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) [1,  8000] loss: 0.519
(pid=4804) [6,  2000] loss: 1.592
Result for DEFAULT_61189_00002:
  accuracy: 0.4243
  date: 2021-05-17_12-25-51
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.5814303705215453
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 272.70516300201416
  time_this_iter_s: 44.634835958480835
  time_total_s: 272.70516300201416
  timestamp: 1621254351
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.272225736999512
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    |                 |            2 |   32 |    4 | 0.00184559  |         |            |                      |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.58143 |     0.4243 |                    6 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.57512 |     0.416  |                    3 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4805) [4,  2000] loss: 1.572
Result for DEFAULT_61189_00000:
  accuracy: 0.3739
  date: 2021-05-17_12-25-57
  done: false
  experiment_id: 403d4e12547345bfb3e360bfb4fba25f
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 1.7403866697456687
  node_ip: 172.17.0.4
  pid: 4799
  should_checkpoint: true
  time_since_restore: 278.58272409439087
  time_this_iter_s: 278.58272409439087
  time_total_s: 278.58272409439087
  timestamp: 1621254357
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00000'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.2324497041225433
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.58143 |     0.4243 |                    6 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.57512 |     0.416  |                    3 |
| DEFAULT_61189_00004 | RUNNING    |                 |            4 |    8 |   64 | 0.0112633   |         |            |                      |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) [1, 10000] loss: 0.422
(pid=4804) [7,  2000] loss: 1.523
(pid=4805) [4,  4000] loss: 0.765
Result for DEFAULT_61189_00004:
  accuracy: 0.1986
  date: 2021-05-17_12-26-21
  done: false
  experiment_id: 4c7b761f313542a78b9833028d95855a
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 2.12941870598793
  node_ip: 172.17.0.4
  pid: 4800
  should_checkpoint: true
  time_since_restore: 142.00827050209045
  time_this_iter_s: 142.00827050209045
  time_total_s: 142.00827050209045
  timestamp: 1621254381
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00004'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.58143 |     0.4243 |                    6 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.57512 |     0.416  |                    3 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.12942 |     0.1986 |                    1 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [2,  2000] loss: 1.684
Result for DEFAULT_61189_00002:
  accuracy: 0.4472
  date: 2021-05-17_12-26-34
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.5020601199150085
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 315.3052477836609
  time_this_iter_s: 42.60008478164673
  time_total_s: 315.3052477836609
  timestamp: 1621254394
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.7205610649108887 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.50206 |     0.4472 |                    7 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.57512 |     0.416  |                    3 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.12942 |     0.1986 |                    1 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [2,  4000] loss: 0.826
Result for DEFAULT_61189_00003:
  accuracy: 0.4584
  date: 2021-05-17_12-26-46
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.4873164200782776
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 327.3121566772461
  time_this_iter_s: 80.78632974624634
  time_total_s: 327.3121566772461
  timestamp: 1621254406
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.50206 |     0.4472 |                    7 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.48732 |     0.4584 |                    4 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.12942 |     0.1986 |                    1 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4800) [2,  2000] loss: 2.195
(pid=4804) [8,  2000] loss: 1.474
(pid=4799) [2,  6000] loss: 0.542
(pid=4800) [2,  4000] loss: 1.095
(pid=4805) [5,  2000] loss: 1.463
Result for DEFAULT_61189_00002:
  accuracy: 0.4782
  date: 2021-05-17_12-27-18
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.4372823927879335
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 358.95592212677
  time_this_iter_s: 43.65067434310913
  time_total_s: 358.95592212677
  timestamp: 1621254438
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.43728 |     0.4782 |                    8 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.48732 |     0.4584 |                    4 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.12942 |     0.1986 |                    1 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [2,  8000] loss: 0.409
(pid=4800) [2,  6000] loss: 0.731
(pid=4805) [5,  4000] loss: 0.725
(pid=4804) [9,  2000] loss: 1.421
(pid=4799) [2, 10000] loss: 0.335
(pid=4800) [2,  8000] loss: 0.576
Result for DEFAULT_61189_00002:
  accuracy: 0.4955
  date: 2021-05-17_12-28-02
  done: false
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 9
  loss: 1.3872015606880188
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 403.1134252548218
  time_this_iter_s: 44.15750312805176
  time_total_s: 403.1134252548218
  timestamp: 1621254482
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '61189_00002'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.3872  |     0.4955 |                    9 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.48732 |     0.4584 |                    4 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.12942 |     0.1986 |                    1 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00003:
  accuracy: 0.4867
  date: 2021-05-17_12-28-02
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.40901914396286
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 403.7404441833496
  time_this_iter_s: 76.42828750610352
  time_total_s: 403.7404441833496
  timestamp: 1621254482
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00003'

(pid=4799) [2, 12000] loss: 0.269
(pid=4800) [2, 10000] loss: 0.460
(pid=4805) [6,  2000] loss: 1.402
(pid=4804) [10,  2000] loss: 1.373
(pid=4799) [2, 14000] loss: 0.228
Result for DEFAULT_61189_00004:
  accuracy: 0.1098
  date: 2021-05-17_12-28-45
  done: true
  experiment_id: 4c7b761f313542a78b9833028d95855a
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 2.29596685628891
  node_ip: 172.17.0.4
  pid: 4800
  should_checkpoint: true
  time_since_restore: 286.11599016189575
  time_this_iter_s: 144.1077196598053
  time_total_s: 286.11599016189575
  timestamp: 1621254525
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00004'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -2.012425883293152 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00002 | RUNNING    | 172.17.0.4:4804 |           16 |   32 |   64 | 0.00028312  | 1.3872  |     0.4955 |                    9 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.40902 |     0.4867 |                    5 |
| DEFAULT_61189_00004 | RUNNING    | 172.17.0.4:4800 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00005 | PENDING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | PENDING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) Files already downloaded and verified
(pid=4803) Files already downloaded and verified
Result for DEFAULT_61189_00002:
  accuracy: 0.4995
  date: 2021-05-17_12-28-50
  done: true
  experiment_id: 0f4bfd9f547c440b8db19b55c493ada7
  hostname: ebeeb9c51d35
  iterations_since_restore: 10
  loss: 1.382101783657074
  node_ip: 172.17.0.4
  pid: 4804
  should_checkpoint: true
  time_since_restore: 451.25868105888367
  time_this_iter_s: 48.14525580406189
  time_total_s: 451.25868105888367
  timestamp: 1621254530
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '61189_00002'

(pid=4801) Files already downloaded and verified
(pid=4801) Files already downloaded and verified
(pid=4805) [6,  4000] loss: 0.674
(pid=4799) [2, 16000] loss: 0.198
(pid=4803) [1,  2000] loss: 2.290
Result for DEFAULT_61189_00003:
  accuracy: 0.514
  date: 2021-05-17_12-29-18
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.3475550504684448
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 479.27017068862915
  time_this_iter_s: 75.52972650527954
  time_total_s: 479.27017068862915
  timestamp: 1621254558
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -2.012425883293152 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.74039 |     0.3739 |                    1 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.34756 |     0.514  |                    6 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [1,  2000] loss: 2.145
(pid=4799) [2, 18000] loss: 0.179
(pid=4803) [1,  4000] loss: 1.085
(pid=4799) [2, 20000] loss: 0.159
(pid=4801) [1,  4000] loss: 0.911
(pid=4805) [7,  2000] loss: 1.326
(pid=4803) [1,  6000] loss: 0.683
(pid=4801) [1,  6000] loss: 0.558
(pid=4805) [7,  4000] loss: 0.653
(pid=4803) [1,  8000] loss: 0.493
(pid=4801) [1,  8000] loss: 0.394
Result for DEFAULT_61189_00000:
  accuracy: 0.4221
  date: 2021-05-17_12-30-19
  done: false
  experiment_id: 403d4e12547345bfb3e360bfb4fba25f
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.5694100677723064
  node_ip: 172.17.0.4
  pid: 4799
  should_checkpoint: true
  time_since_restore: 540.8643796443939
  time_this_iter_s: 262.28165555000305
  time_total_s: 540.8643796443939
  timestamp: 1621254619
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00000'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.34756 |     0.514  |                    6 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00003:
  accuracy: 0.5319
  date: 2021-05-17_12-30-26
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.3008135415792466
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 547.2975535392761
  time_this_iter_s: 68.02738285064697
  time_total_s: 547.2975535392761
  timestamp: 1621254626
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1926736712455748
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.30081 |     0.5319 |                    7 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    |                 |            4 |   64 |   32 | 0.000998733 |         |            |                      |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [1, 10000] loss: 0.381
(pid=4801) [1, 10000] loss: 0.306
(pid=4799) [3,  2000] loss: 1.563
(pid=4805) [8,  2000] loss: 1.261
(pid=4803) [1, 12000] loss: 0.305
Result for DEFAULT_61189_00006:
  accuracy: 0.4535
  date: 2021-05-17_12-31-00
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 1.515771720123291
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 128.15965723991394
  time_this_iter_s: 128.15965723991394
  time_total_s: 128.15965723991394
  timestamp: 1621254660
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.4372823927879335 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1610461886167522
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.30081 |     0.5319 |                    7 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.51577 |     0.4535 |                    1 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [3,  4000] loss: 0.786
(pid=4803) [1, 14000] loss: 0.254
(pid=4805) [8,  4000] loss: 0.625
(pid=4801) [2,  2000] loss: 1.432
(pid=4799) [3,  6000] loss: 0.517
(pid=4803) [1, 16000] loss: 0.215
Result for DEFAULT_61189_00003:
  accuracy: 0.541
  date: 2021-05-17_12-31-40
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.2750375418901443
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 620.9241535663605
  time_this_iter_s: 73.62660002708435
  time_total_s: 620.9241535663605
  timestamp: 1621254700
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1610461886167522
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.27504 |     0.541  |                    8 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.51577 |     0.4535 |                    1 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [3,  8000] loss: 0.395
(pid=4801) [2,  4000] loss: 0.712
(pid=4803) [1, 18000] loss: 0.185
(pid=4805) [9,  2000] loss: 1.214
(pid=4801) [2,  6000] loss: 0.463
(pid=4799) [3, 10000] loss: 0.330
(pid=4803) [1, 20000] loss: 0.163
(pid=4805) [9,  4000] loss: 0.595
(pid=4801) [2,  8000] loss: 0.342
(pid=4799) [3, 12000] loss: 0.263
(pid=4801) [2, 10000] loss: 0.271
(pid=4799) [3, 14000] loss: 0.228
Result for DEFAULT_61189_00003:
  accuracy: 0.5496
  date: 2021-05-17_12-32-52
  done: false
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 9
  loss: 1.2579754381895065
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 692.7824611663818
  time_this_iter_s: 71.85830760002136
  time_total_s: 692.7824611663818
  timestamp: 1621254772
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.1610461886167522
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.25798 |     0.5496 |                    9 |
| DEFAULT_61189_00005 | RUNNING    |                 |            2 |    8 |  256 | 0.000214909 |         |            |                      |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.51577 |     0.4535 |                    1 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00005:
  accuracy: 0.4001
  date: 2021-05-17_12-33-00
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 1.603653232383728
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 253.12474060058594
  time_this_iter_s: 253.12474060058594
  time_total_s: 253.12474060058594
  timestamp: 1621254780
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.896151064133644 | Iter 1.000: -2.12941870598793
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.25798 |     0.5496 |                    9 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.51577 |     0.4535 |                    1 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [3, 16000] loss: 0.200
Result for DEFAULT_61189_00006:
  accuracy: 0.5207
  date: 2021-05-17_12-33-11
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.325894591587782
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 259.8798174858093
  time_this_iter_s: 131.72016024589539
  time_total_s: 259.8798174858093
  timestamp: 1621254791
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.7798762449741363 | Iter 1.000: -2.12941870598793
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.25798 |     0.5496 |                    9 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.32589 |     0.5207 |                    2 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4805) [10,  2000] loss: 1.167
(pid=4803) [2,  2000] loss: 1.606
(pid=4799) [3, 18000] loss: 0.179
(pid=4801) [3,  2000] loss: 1.286
(pid=4805) [10,  4000] loss: 0.582
(pid=4803) [2,  4000] loss: 0.800
(pid=4799) [3, 20000] loss: 0.156
(pid=4801) [3,  4000] loss: 0.634
Result for DEFAULT_61189_00003:
  accuracy: 0.5736
  date: 2021-05-17_12-34-03
  done: true
  experiment_id: f435e170717a454c827bc47994eee63b
  hostname: ebeeb9c51d35
  iterations_since_restore: 10
  loss: 1.1951737923622132
  node_ip: 172.17.0.4
  pid: 4805
  should_checkpoint: true
  time_since_restore: 764.4231495857239
  time_this_iter_s: 71.64068841934204
  time_total_s: 764.4231495857239
  timestamp: 1621254843
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '61189_00003'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.7798762449741363 | Iter 1.000: -2.12941870598793
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.56941 |     0.4221 |                    2 |
| DEFAULT_61189_00003 | RUNNING    | 172.17.0.4:4805 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.32589 |     0.5207 |                    2 |
| DEFAULT_61189_00007 | PENDING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4802) Files already downloaded and verified
(pid=4803) [2,  6000] loss: 0.520
(pid=4802) Files already downloaded and verified
(pid=4801) [3,  6000] loss: 0.423
(pid=4803) [2,  8000] loss: 0.384
(pid=4802) [1,  2000] loss: 2.128
Result for DEFAULT_61189_00000:
  accuracy: 0.4017
  date: 2021-05-17_12-34-38
  done: false
  experiment_id: 403d4e12547345bfb3e360bfb4fba25f
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.6570855555281974
  node_ip: 172.17.0.4
  pid: 4799
  should_checkpoint: true
  time_since_restore: 799.2564842700958
  time_this_iter_s: 258.3921046257019
  time_total_s: 799.2564842700958
  timestamp: 1621254878
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00000'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.7798762449741363 | Iter 1.000: -2.12941870598793
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.32589 |     0.5207 |                    2 |
| DEFAULT_61189_00007 | RUNNING    |                 |            8 |   64 |   64 | 0.00120993  |         |            |                      |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [3,  8000] loss: 0.311
(pid=4803) [2, 10000] loss: 0.302
(pid=4802) [1,  4000] loss: 0.876
(pid=4799) [4,  2000] loss: 1.545
(pid=4801) [3, 10000] loss: 0.250
(pid=4803) [2, 12000] loss: 0.247
Result for DEFAULT_61189_00007:
  accuracy: 0.3946
  date: 2021-05-17_12-35-18
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 1.6429293813228607
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 72.35904288291931
  time_this_iter_s: 72.35904288291931
  time_total_s: 72.35904288291931
  timestamp: 1621254918
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.7798762449741363 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.32589 |     0.5207 |                    2 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.64293 |     0.3946 |                    1 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [4,  4000] loss: 0.781
Result for DEFAULT_61189_00006:
  accuracy: 0.5596
  date: 2021-05-17_12-35-21
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.2459865901648999
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 389.81822538375854
  time_this_iter_s: 129.93840789794922
  time_total_s: 389.81822538375854
  timestamp: 1621254921
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00006'

(pid=4803) [2, 14000] loss: 0.212
(pid=4799) [4,  6000] loss: 0.510
(pid=4802) [2,  2000] loss: 1.545
(pid=4801) [4,  2000] loss: 1.164
(pid=4803) [2, 16000] loss: 0.183
(pid=4799) [4,  8000] loss: 0.394
(pid=4801) [4,  4000] loss: 0.594
(pid=4803) [2, 18000] loss: 0.160
(pid=4802) [2,  4000] loss: 0.741
(pid=4799) [4, 10000] loss: 0.314
(pid=4801) [4,  6000] loss: 0.395
(pid=4803) [2, 20000] loss: 0.142
Result for DEFAULT_61189_00007:
  accuracy: 0.503
  date: 2021-05-17_12-36-29
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.3987183806180954
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 143.7843976020813
  time_this_iter_s: 71.42535471916199
  time_total_s: 143.7843976020813
  timestamp: 1621254989
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.6746431563732214 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.60365 |     0.4001 |                    1 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.24599 |     0.5596 |                    3 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.39872 |     0.503  |                    2 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4799) [4, 12000] loss: 0.259
(pid=4801) [4,  8000] loss: 0.294
(pid=4802) [3,  2000] loss: 1.381
(pid=4799) [4, 14000] loss: 0.226
(pid=4801) [4, 10000] loss: 0.235
Result for DEFAULT_61189_00005:
  accuracy: 0.4874
  date: 2021-05-17_12-37-07
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.3904137761592865
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 499.89836716651917
  time_this_iter_s: 246.77362656593323
  time_total_s: 499.89836716651917
  timestamp: 1621255027
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.603938742494583 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.24599 |     0.5596 |                    3 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.39872 |     0.503  |                    2 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4802) [3,  4000] loss: 0.666
(pid=4799) [4, 16000] loss: 0.198
Result for DEFAULT_61189_00006:
  accuracy: 0.5726
  date: 2021-05-17_12-37-29
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.1961397072136402
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 517.673437833786
  time_this_iter_s: 127.85521245002747
  time_total_s: 517.673437833786
  timestamp: 1621255049
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.19614 |     0.5726 |                    4 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.39872 |     0.503  |                    2 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [3,  2000] loss: 1.404
(pid=4799) [4, 18000] loss: 0.174
Result for DEFAULT_61189_00007:
  accuracy: 0.5354
  date: 2021-05-17_12-37-41
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.3133668680429458
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 216.08084869384766
  time_this_iter_s: 72.29645109176636
  time_total_s: 216.08084869384766
  timestamp: 1621255061
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.65709 |     0.4017 |                    3 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.19614 |     0.5726 |                    4 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.31337 |     0.5354 |                    3 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [5,  2000] loss: 1.108
(pid=4803) [3,  4000] loss: 0.706
(pid=4799) [4, 20000] loss: 0.158
(pid=4802) [4,  2000] loss: 1.255
(pid=4803) [3,  6000] loss: 0.458
(pid=4801) [5,  4000] loss: 0.560
(pid=4802) [4,  4000] loss: 0.617
(pid=4803) [3,  8000] loss: 0.352
(pid=4801) [5,  6000] loss: 0.367
Result for DEFAULT_61189_00000:
  accuracy: 0.4211
  date: 2021-05-17_12-38-37
  done: true
  experiment_id: 403d4e12547345bfb3e360bfb4fba25f
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.6108263819852378
  node_ip: 172.17.0.4
  pid: 4799
  should_checkpoint: true
  time_since_restore: 1038.1591680049896
  time_this_iter_s: 238.9026837348938
  time_total_s: 1038.1591680049896
  timestamp: 1621255117
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00000'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.5490714010317577 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | RUNNING    | 172.17.0.4:4799 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.19614 |     0.5726 |                    4 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.31337 |     0.5354 |                    3 |
| DEFAULT_61189_00008 | PENDING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=11624) Files already downloaded and verified
(pid=11624) Files already downloaded and verified
(pid=4803) [3, 10000] loss: 0.267
(pid=4801) [5,  8000] loss: 0.285
Result for DEFAULT_61189_00007:
  accuracy: 0.5575
  date: 2021-05-17_12-38-51
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.2580487357139587
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 286.0748805999756
  time_this_iter_s: 69.99403190612793
  time_total_s: 286.0748805999756
  timestamp: 1621255131
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 8.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.19614 |     0.5726 |                    4 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.25805 |     0.5575 |                    4 |
| DEFAULT_61189_00008 | RUNNING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=11624) [1,  2000] loss: 2.146
(pid=4803) [3, 12000] loss: 0.228
(pid=4801) [5, 10000] loss: 0.222
(pid=4802) [5,  2000] loss: 1.178
(pid=11624) [1,  4000] loss: 1.010
(pid=4803) [3, 14000] loss: 0.195
Result for DEFAULT_61189_00006:
  accuracy: 0.5746
  date: 2021-05-17_12-39-31
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.210697157922387
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 639.4823579788208
  time_this_iter_s: 121.80892014503479
  time_total_s: 639.4823579788208
  timestamp: 1621255171
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.2107  |     0.5746 |                    5 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.25805 |     0.5575 |                    4 |
| DEFAULT_61189_00008 | RUNNING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4802) [5,  4000] loss: 0.581
(pid=11624) [1,  6000] loss: 0.667
(pid=4803) [3, 16000] loss: 0.168
(pid=4801) [6,  2000] loss: 1.063
(pid=11624) [1,  8000] loss: 0.493
Result for DEFAULT_61189_00007:
  accuracy: 0.5849
  date: 2021-05-17_12-40-04
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.1710419122934341
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 358.6585509777069
  time_this_iter_s: 72.58367037773132
  time_total_s: 358.6585509777069
  timestamp: 1621255204
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.9349026878667994
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.2107  |     0.5746 |                    5 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.17104 |     0.5849 |                    5 |
| DEFAULT_61189_00008 | RUNNING    |                 |            4 |    4 |  128 | 0.00764251  |         |            |                      |
| DEFAULT_61189_00009 | PENDING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [3, 18000] loss: 0.151
(pid=4801) [6,  4000] loss: 0.540
(pid=11624) [1, 10000] loss: 0.394
(pid=4802) [6,  2000] loss: 1.110
(pid=4803) [3, 20000] loss: 0.133
(pid=4801) [6,  6000] loss: 0.355
Result for DEFAULT_61189_00008:
  accuracy: 0.2448
  date: 2021-05-17_12-40-48
  done: true
  experiment_id: f98598b2d4f7401e9afdb5fbaf5b8cbf
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 2.002202976179123
  node_ip: 172.17.0.4
  pid: 11624
  should_checkpoint: true
  time_since_restore: 127.09166216850281
  time_this_iter_s: 127.09166216850281
  time_total_s: 127.09166216850281
  timestamp: 1621255248
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00008'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -2.002202976179123
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.39041 |     0.4874 |                    2 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.2107  |     0.5746 |                    5 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.17104 |     0.5849 |                    5 |
| DEFAULT_61189_00008 | RUNNING    | 172.17.0.4:11624 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | PENDING    |                  |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=12441) Files already downloaded and verified
(pid=4802) [6,  4000] loss: 0.562
(pid=12441) Files already downloaded and verified
(pid=4801) [6,  8000] loss: 0.271
Result for DEFAULT_61189_00005:
  accuracy: 0.5099
  date: 2021-05-17_12-41-13
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.33840386143215
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 746.1071588993073
  time_this_iter_s: 246.2087917327881
  time_total_s: 746.1071588993073
  timestamp: 1621255273
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 9.0/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -2.002202976179123
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.2107  |     0.5746 |                    5 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802 |            8 |   64 |   64 | 0.00120993  | 1.17104 |     0.5849 |                    5 |
| DEFAULT_61189_00009 | RUNNING    |                 |           16 |    8 |  256 | 0.00387042  |         |            |                      |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [6, 10000] loss: 0.212
Result for DEFAULT_61189_00007:
  accuracy: 0.5971
  date: 2021-05-17_12-41-17
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.1495505164146422
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 431.2005603313446
  time_this_iter_s: 72.5420093536377
  time_total_s: 431.2005603313446
  timestamp: 1621255277
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00007'

(pid=12441) [1,  2000] loss: 1.869
(pid=4803) [4,  2000] loss: 1.282
Result for DEFAULT_61189_00009:
  accuracy: 0.4292
  date: 2021-05-17_12-41-34
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 1
  loss: 1.5647239219665527
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 44.01426815986633
  time_this_iter_s: 44.01426815986633
  time_total_s: 44.01426815986633
  timestamp: 1621255294
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.5694100677723064 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.2107  |     0.5746 |                    5 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.14955 |     0.5971 |                    6 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.56472 |     0.4292 |                    1 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00006:
  accuracy: 0.5789
  date: 2021-05-17_12-41-38
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.2016486544713378
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 766.2602305412292
  time_this_iter_s: 126.77787256240845
  time_total_s: 766.2602305412292
  timestamp: 1621255298
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00006'

(pid=4802) [7,  2000] loss: 1.062
(pid=4803) [4,  4000] loss: 0.631
(pid=4801) [7,  2000] loss: 1.019
(pid=12441) [2,  2000] loss: 1.447
(pid=4802) [7,  4000] loss: 0.534
(pid=4803) [4,  6000] loss: 0.437
Result for DEFAULT_61189_00009:
  accuracy: 0.5025
  date: 2021-05-17_12-42-17
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 2
  loss: 1.3989874996185303
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 87.00301432609558
  time_this_iter_s: 42.98874616622925
  time_total_s: 87.00301432609558
  timestamp: 1621255337
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.1/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.20165 |     0.5789 |                    6 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.14955 |     0.5971 |                    6 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.39899 |     0.5025 |                    2 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [7,  4000] loss: 0.511
Result for DEFAULT_61189_00007:
  accuracy: 0.5856
  date: 2021-05-17_12-42-31
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.1952720649719237
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 506.0727801322937
  time_this_iter_s: 74.8722198009491
  time_total_s: 506.0727801322937
  timestamp: 1621255351
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.20165 |     0.5789 |                    6 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.19527 |     0.5856 |                    7 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.39899 |     0.5025 |                    2 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [4,  8000] loss: 0.319
(pid=4801) [7,  6000] loss: 0.351
(pid=12441) [3,  2000] loss: 1.322
(pid=4803) [4, 10000] loss: 0.261
(pid=4802) [8,  2000] loss: 1.029
Result for DEFAULT_61189_00009:
  accuracy: 0.5436
  date: 2021-05-17_12-43-02
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 3
  loss: 1.2798112030029296
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 132.02003479003906
  time_this_iter_s: 45.01702046394348
  time_total_s: 132.02003479003906
  timestamp: 1621255382
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.356159967339039 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.20165 |     0.5789 |                    6 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.19527 |     0.5856 |                    7 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.27981 |     0.5436 |                    3 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [7,  8000] loss: 0.259
(pid=4803) [4, 12000] loss: 0.220
(pid=4802) [8,  4000] loss: 0.518
(pid=4801) [7, 10000] loss: 0.205
(pid=12441) [4,  2000] loss: 1.250
(pid=4803) [4, 14000] loss: 0.186
Result for DEFAULT_61189_00007:
  accuracy: 0.6028
  date: 2021-05-17_12-43-42
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.1159002369403839
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 576.4117579460144
  time_this_iter_s: 70.3389778137207
  time_total_s: 576.4117579460144
  timestamp: 1621255422
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.4873164200782776 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.20165 |     0.5789 |                    6 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.1159  |     0.6028 |                    8 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.27981 |     0.5436 |                    3 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00009:
  accuracy: 0.5679
  date: 2021-05-17_12-43-44
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.2381296847343446
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 173.0951099395752
  time_this_iter_s: 41.07507514953613
  time_total_s: 173.0951099395752
  timestamp: 1621255424
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00009'

Result for DEFAULT_61189_00006:
  accuracy: 0.6019
  date: 2021-05-17_12-43-48
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.1383088717967271
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 896.3186950683594
  time_this_iter_s: 130.05846452713013
  time_total_s: 896.3186950683594
  timestamp: 1621255428
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3726825778961183 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.1159  |     0.6028 |                    8 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.23813 |     0.5679 |                    4 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [4, 16000] loss: 0.161
(pid=4802) [9,  2000] loss: 0.964
(pid=4801) [8,  2000] loss: 0.970
(pid=12441) [5,  2000] loss: 1.221
(pid=4803) [4, 18000] loss: 0.146
Result for DEFAULT_61189_00009:
  accuracy: 0.5455
  date: 2021-05-17_12-44-29
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.2983551637649535
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 218.95898032188416
  time_this_iter_s: 45.86387038230896
  time_total_s: 218.95898032188416
  timestamp: 1621255469
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3726825778961183 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.1159  |     0.6028 |                    8 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.29836 |     0.5455 |                    5 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4802) [9,  4000] loss: 0.505
(pid=4801) [8,  4000] loss: 0.501
(pid=4803) [4, 20000] loss: 0.131
(pid=4801) [8,  6000] loss: 0.336
Result for DEFAULT_61189_00007:
  accuracy: 0.5981
  date: 2021-05-17_12-44-57
  done: false
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 9
  loss: 1.1250858846306802
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 651.4575672149658
  time_this_iter_s: 75.04580926895142
  time_total_s: 651.4575672149658
  timestamp: 1621255497
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3726825778961183 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.12509 |     0.5981 |                    9 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.29836 |     0.5455 |                    5 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=12441) [6,  2000] loss: 1.185
Result for DEFAULT_61189_00009:
  accuracy: 0.567
  date: 2021-05-17_12-45-11
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.2448465644836426
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 260.4495847225189
  time_this_iter_s: 41.490604400634766
  time_total_s: 260.4495847225189
  timestamp: 1621255511
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3726825778961183 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.3384  |     0.5099 |                    3 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.12509 |     0.5981 |                    9 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.24485 |     0.567  |                    6 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00005:
  accuracy: 0.5268
  date: 2021-05-17_12-45-17
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 4
  loss: 1.3136553026117384
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 989.433837890625
  time_this_iter_s: 243.32667899131775
  time_total_s: 989.433837890625
  timestamp: 1621255517
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.12509 |     0.5981 |                    9 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.24485 |     0.567  |                    6 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [8,  8000] loss: 0.252
(pid=4802) [10,  2000] loss: 0.951
(pid=4801) [8, 10000] loss: 0.197
(pid=4803) [5,  2000] loss: 1.244
(pid=12441) [7,  2000] loss: 1.160
(pid=4802) [10,  4000] loss: 0.480
Result for DEFAULT_61189_00009:
  accuracy: 0.5555
  date: 2021-05-17_12-45-54
  done: false
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.2535951126098632
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 303.4837951660156
  time_this_iter_s: 43.034210443496704
  time_total_s: 303.4837951660156
  timestamp: 1621255554
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 9.3/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750375418901443 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.13831 |     0.6019 |                    7 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.12509 |     0.5981 |                    9 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.2536  |     0.5555 |                    7 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [5,  4000] loss: 0.628
Result for DEFAULT_61189_00006:
  accuracy: 0.6041
  date: 2021-05-17_12-46-00
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.1655468902409076
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 1028.7229526042938
  time_this_iter_s: 132.40425753593445
  time_total_s: 1028.7229526042938
  timestamp: 1621255560
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 9.4/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.220292216065526 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.16555 |     0.6041 |                    8 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.12509 |     0.5981 |                    9 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.2536  |     0.5555 |                    7 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_61189_00007:
  accuracy: 0.6025
  date: 2021-05-17_12-46-11
  done: true
  experiment_id: a23ba9d8db374766b82cad29b8814caa
  hostname: ebeeb9c51d35
  iterations_since_restore: 10
  loss: 1.1581312307357787
  node_ip: 172.17.0.4
  pid: 4802
  should_checkpoint: true
  time_since_restore: 725.1176142692566
  time_this_iter_s: 73.66004705429077
  time_total_s: 725.1176142692566
  timestamp: 1621255571
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '61189_00007'

== Status ==
Memory usage on this node: 9.4/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.220292216065526 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.16555 |     0.6041 |                    8 |
| DEFAULT_61189_00007 | RUNNING    | 172.17.0.4:4802  |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.2536  |     0.5555 |                    7 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [5,  6000] loss: 0.419
(pid=4801) [9,  2000] loss: 0.939
(pid=12441) [8,  2000] loss: 1.146
Result for DEFAULT_61189_00009:
  accuracy: 0.576
  date: 2021-05-17_12-46-33
  done: true
  experiment_id: ba8f20dd25e2465392a244a5ff342be2
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.224115501356125
  node_ip: 172.17.0.4
  pid: 12441
  should_checkpoint: true
  time_since_restore: 343.027822971344
  time_this_iter_s: 39.54402780532837
  time_total_s: 343.027822971344
  timestamp: 1621255593
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00009'

== Status ==
Memory usage on this node: 8.5/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc              |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803  |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801  |            4 |   64 |   32 | 0.000998733 | 1.16555 |     0.6041 |                    8 |
| DEFAULT_61189_00009 | RUNNING    | 172.17.0.4:12441 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
| DEFAULT_61189_00000 | TERMINATED |                  |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                  |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                  |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                  |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                  |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00007 | TERMINATED |                  |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                  |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
+---------------------+------------+------------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [5,  8000] loss: 0.317
(pid=4801) [9,  4000] loss: 0.483
(pid=4803) [5, 10000] loss: 0.249
(pid=4801) [9,  6000] loss: 0.321
(pid=4801) [9,  8000] loss: 0.248
(pid=4803) [5, 12000] loss: 0.206
(pid=4803) [5, 14000] loss: 0.180
(pid=4801) [9, 10000] loss: 0.195
Result for DEFAULT_61189_00006:
  accuracy: 0.6129
  date: 2021-05-17_12-47-35
  done: false
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 9
  loss: 1.1447944878730922
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 1123.7585654258728
  time_this_iter_s: 95.03561282157898
  time_total_s: 1123.7585654258728
  timestamp: 1621255655
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 7.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.31366 |     0.5268 |                    4 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.14479 |     0.6129 |                    9 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [5, 16000] loss: 0.154
(pid=4803) [5, 18000] loss: 0.137
(pid=4801) [10,  2000] loss: 0.888
(pid=4801) [10,  4000] loss: 0.475
(pid=4803) [5, 20000] loss: 0.121
(pid=4801) [10,  6000] loss: 0.319
Result for DEFAULT_61189_00005:
  accuracy: 0.5452
  date: 2021-05-17_12-48-32
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 5
  loss: 1.2476676940646023
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1184.944656610489
  time_this_iter_s: 195.5108187198639
  time_total_s: 1184.944656610489
  timestamp: 1621255712
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 7.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.24767 |     0.5452 |                    5 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.14479 |     0.6129 |                    9 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4801) [10,  8000] loss: 0.238
(pid=4803) [6,  2000] loss: 1.184
(pid=4801) [10, 10000] loss: 0.192
(pid=4803) [6,  4000] loss: 0.609
Result for DEFAULT_61189_00006:
  accuracy: 0.5926
  date: 2021-05-17_12-49-04
  done: true
  experiment_id: 6201266fc2cc4f6a859dc6dbfe9a83f3
  hostname: ebeeb9c51d35
  iterations_since_restore: 10
  loss: 1.202866588556394
  node_ip: 172.17.0.4
  pid: 4801
  should_checkpoint: true
  time_since_restore: 1212.863196849823
  time_this_iter_s: 89.1046314239502
  time_total_s: 1212.863196849823
  timestamp: 1621255744
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '61189_00006'

== Status ==
Memory usage on this node: 7.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.24767 |     0.5452 |                    5 |
| DEFAULT_61189_00006 | RUNNING    | 172.17.0.4:4801 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [6,  6000] loss: 0.409
(pid=4803) [6,  8000] loss: 0.292
(pid=4803) [6, 10000] loss: 0.236
(pid=4803) [6, 12000] loss: 0.206
(pid=4803) [6, 14000] loss: 0.174
(pid=4803) [6, 16000] loss: 0.150
(pid=4803) [6, 18000] loss: 0.134
(pid=4803) [6, 20000] loss: 0.120
Result for DEFAULT_61189_00005:
  accuracy: 0.5855
  date: 2021-05-17_12-50-20
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 6
  loss: 1.1616042586140334
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1292.5585188865662
  time_this_iter_s: 107.61386227607727
  time_total_s: 1292.5585188865662
  timestamp: 1621255820
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 6.8/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.1616  |     0.5855 |                    6 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00006 | TERMINATED |                 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [7,  2000] loss: 1.159
(pid=4803) [7,  4000] loss: 0.590
(pid=4803) [7,  6000] loss: 0.391
(pid=4803) [7,  8000] loss: 0.295
(pid=4803) [7, 10000] loss: 0.234
(pid=4803) [7, 12000] loss: 0.191
(pid=4803) [7, 14000] loss: 0.166
(pid=4803) [7, 16000] loss: 0.148
(pid=4803) [7, 18000] loss: 0.133
(pid=4803) [7, 20000] loss: 0.116
Result for DEFAULT_61189_00005:
  accuracy: 0.5844
  date: 2021-05-17_12-51-52
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 7
  loss: 1.1713121899626218
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1384.9634327888489
  time_this_iter_s: 92.40491390228271
  time_total_s: 1384.9634327888489
  timestamp: 1621255912
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 6.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.224115501356125 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.17131 |     0.5844 |                    7 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00006 | TERMINATED |                 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [8,  2000] loss: 1.161
(pid=4803) [8,  4000] loss: 0.580
(pid=4803) [8,  6000] loss: 0.387
(pid=4803) [8,  8000] loss: 0.277
(pid=4803) [8, 10000] loss: 0.228
(pid=4803) [8, 12000] loss: 0.191
(pid=4803) [8, 14000] loss: 0.166
(pid=4803) [8, 16000] loss: 0.145
(pid=4803) [8, 18000] loss: 0.126
(pid=4803) [8, 20000] loss: 0.113
Result for DEFAULT_61189_00005:
  accuracy: 0.5716
  date: 2021-05-17_12-53-23
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 8
  loss: 1.1928544228610931
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1476.0283370018005
  time_this_iter_s: 91.06490421295166
  time_total_s: 1476.0283370018005
  timestamp: 1621256003
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 6.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2084849621086091 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.19285 |     0.5716 |                    8 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00006 | TERMINATED |                 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [9,  2000] loss: 1.123
(pid=4803) [9,  4000] loss: 0.562
(pid=4803) [9,  6000] loss: 0.368
(pid=4803) [9,  8000] loss: 0.280
(pid=4803) [9, 10000] loss: 0.225
(pid=4803) [9, 12000] loss: 0.187
(pid=4803) [9, 14000] loss: 0.163
(pid=4803) [9, 16000] loss: 0.142
(pid=4803) [9, 18000] loss: 0.127
(pid=4803) [9, 20000] loss: 0.112
Result for DEFAULT_61189_00005:
  accuracy: 0.5979
  date: 2021-05-17_12-54-55
  done: false
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 9
  loss: 1.1343754458556883
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1567.9582397937775
  time_this_iter_s: 91.92990279197693
  time_total_s: 1567.9582397937775
  timestamp: 1621256095
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 6.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2084849621086091 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.13438 |     0.5979 |                    9 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00006 | TERMINATED |                 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=4803) [10,  2000] loss: 1.095
(pid=4803) [10,  4000] loss: 0.554
(pid=4803) [10,  6000] loss: 0.371
(pid=4803) [10,  8000] loss: 0.278
(pid=4803) [10, 10000] loss: 0.219
(pid=4803) [10, 12000] loss: 0.184
(pid=4803) [10, 14000] loss: 0.162
(pid=4803) [10, 16000] loss: 0.138
(pid=4803) [10, 18000] loss: 0.124
(pid=4803) [10, 20000] loss: 0.114
Result for DEFAULT_61189_00005:
  accuracy: 0.5857
  date: 2021-05-17_12-56-26
  done: true
  experiment_id: 44f9a4bd23754be5b9498d6d2b9edc51
  hostname: ebeeb9c51d35
  iterations_since_restore: 10
  loss: 1.1718126490584109
  node_ip: 172.17.0.4
  pid: 4803
  should_checkpoint: true
  time_since_restore: 1659.0728154182434
  time_this_iter_s: 91.11457562446594
  time_total_s: 1659.0728154182434
  timestamp: 1621256186
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '61189_00005'

== Status ==
Memory usage on this node: 6.9/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2084849621086091 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00005 | RUNNING    | 172.17.0.4:4803 |            2 |    8 |  256 | 0.000214909 | 1.17181 |     0.5857 |                   10 |
| DEFAULT_61189_00000 | TERMINATED |                 |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |                 |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |                 |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |                 |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |                 |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00006 | TERMINATED |                 |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |                 |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |                 |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |                 |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Memory usage on this node: 6.8/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2084849621086091 | Iter 4.000: -1.3136553026117384 | Iter 2.000: -1.4841987836954185 | Iter 1.000: -1.8712948229623958
Resources requested: 0/8 CPUs, 0/2 GPUs, 0.0/15.53 GiB heap, 0.0/7.76 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-05-17_12-21-17
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc   |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_61189_00000 | TERMINATED |       |            2 |   32 |    4 | 0.00184559  | 1.61083 |     0.4211 |                    4 |
| DEFAULT_61189_00001 | TERMINATED |       |            4 |   16 |  256 | 0.055604    | 2.32989 |     0.1026 |                    1 |
| DEFAULT_61189_00002 | TERMINATED |       |           16 |   32 |   64 | 0.00028312  | 1.3821  |     0.4995 |                   10 |
| DEFAULT_61189_00003 | TERMINATED |       |            8 |  128 |  128 | 0.000251858 | 1.19517 |     0.5736 |                   10 |
| DEFAULT_61189_00004 | TERMINATED |       |            4 |    8 |   64 | 0.0112633   | 2.29597 |     0.1098 |                    2 |
| DEFAULT_61189_00005 | TERMINATED |       |            2 |    8 |  256 | 0.000214909 | 1.17181 |     0.5857 |                   10 |
| DEFAULT_61189_00006 | TERMINATED |       |            4 |   64 |   32 | 0.000998733 | 1.20287 |     0.5926 |                   10 |
| DEFAULT_61189_00007 | TERMINATED |       |            8 |   64 |   64 | 0.00120993  | 1.15813 |     0.6025 |                   10 |
| DEFAULT_61189_00008 | TERMINATED |       |            4 |    4 |  128 | 0.00764251  | 2.0022  |     0.2448 |                    1 |
| DEFAULT_61189_00009 | TERMINATED |       |           16 |    8 |  256 | 0.00387042  | 1.22412 |     0.576  |                    8 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+


Best trial config: {'l1': 64, 'l2': 64, 'lr': 0.0012099282775404714, 'batch_size': 8}
Best trial final validation loss: 1.1581312307357787
Best trial final validation accuracy: 0.6025
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.5998

If you run the code, an example output could look like this:

Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
| ... |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
| ... |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
| ... |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
| ... |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
| ... |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
| ... |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
| ... |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
| ... |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
| ... |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+


Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806

Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.

So that’s it! You can now tune the parameters of your PyTorch models.