Welcome to Butler’s documentation!

class butler.Butler(keywords=(), keep_keywords=True, setup_file='setup.json', session_parent_dir='C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\experiments', output_variable_name='', data_variables=(), img_files=(), fig_files=(), data_files=(), ignore_colours=True, create_new_exp_on_run=False)

Butler collects and organizes experiment data into folders while the experiment is running.

Parameters

  • keywords (str or list or tuple) – List of keywords whose lines will be extracted when printed

  • keep_keywords (bool) – Whether to also save the keywords with the rest of the print line

  • setup_file (str) – Path to the json containing the setup mappings. E.g. {“gripper”: “robotiq 2f85”, …}

  • session_parent_dir (str) – Directory where to save the experiments; default is the butler.py directory

  • output_variable_name (str) – The string name of the variable that contains the data that is otherwise returned by the decorated function. Has to be visible in the scope where the decorated function is called. E.g. self.data_var or just_data_var.

  • data_variables (list or tuple or str or dict[str, list or tuple or np.ndarray or str]) – Sensor output variables. Format: {“source_sensor_name”: {“quantity (e.g. postiion)”: [list, of, values], …}, …}

  • img_files (list[str] or tuple[str]) – List of file paths that will be copied to experiment_i/property_j/imgs every time the function is run.

  • fig_files (list[str] or tuple[str]) – List of file paths that will be copied to experiment_i/property_j/figs every time the function is run.

  • data_files (list[str] or tuple[str]) – List of file paths that will be copied to experiment_i/property_j/data every time the function is run.

  • ignore_colours (bool) – Whether to ignore the special colour characters; in regex: “+](;d+)?m”

  • create_new_exp_on_run (bool) – Whether to create a new experiment_i folder on every run of the function.

static add_measurement_png(png_path)

Adds the png image to be uploaded into the measurement[“png”] slot

Parameters

png_path (str) – Path to the png file as it is on the disk.

static add_object_context(context, override_recommendation=False)

Adds the object’s “maker”, “common_name”, “dataset”, “dataset_id” for one run of the function. Call Butler.add_object_context(context) inside the decorated funtion.

Parameters

  • context (dict[str, str]) – The object context. Format: {“maker”: “ikea”, “common_name”: “hard_yellow_sponge”, “dataset”: “ycb”}. If “dataset_id” is present, “dataset” must also be present.

  • override_recommendation (bool) – Whether to remove the constraint that the context keys have to be one of [“maker”, “common_name”, “dataset”, “dataset_id”.]

static add_tmp_files(file_paths, tmp_file_folder, target_names=None)

Adds files to be copied to the folder: property_j/[data, imgs, figs] when the function decorated by @butler is called.

Parameters

  • file_paths (list[str] or str) – The files to be copied to tmp_file_folder.

  • tmp_file_folder (str) – One of [“data”, “imgs”, “figs”]. The experiment_i/property_j subfolder into which the file_paths files are to be copied.

  • target_names (list[str] or str) – The names that the file_paths names will be copied into. E.g. file_paths=”/tmp/box.png”, target_names=”cheezit.png”

class butler.CustomBytesIO(initial_value='', newline='\n')

BytesIO pipe that copies all prints into itself

write(data)

Write string to file.

Returns the number of characters written, which is always equal to the length of the string.

butler.cache_print(f, *args, **kwargs)

Save the return value of function f & its print outputs

Parameters

  • f (function) – Function whose prints you want saved.

  • args (any) – Classical *args.

  • kwargs (dict) – Classical **kwargs.

Returns

tuple of (function_return, stdout)

Return type

tuple

butler.format_data_variables(data_vars)

Removes unused “sensors” from the data variables.

Parameters

data_vars (tuple[dict]) – Tuple of data variables formatted like a dictionary {“sensor”: {“quantity”: values}}

butler.this_dir()

Returns the directory of the current python file

Returns

Current parent dir of this file

Return type

str

formatting.convert_experiments(interval='__all__')

Format the experiments in the config.experiment_directory folder to dicts.

Parameters

interval (str or list) –

Date interval between which to convert the experiments.

Values:

”__all__”: all experiments - ok if you call this manually and you dont have thousands of properties

[a or None, b or None] - from a to b or all before b or all after a. Format: YYYY_mm_dd_HH_MM_SS

formatting.experiment_to_json(experiment_directory, out_file=None)

Takes a butlered experiment folder and converts it into a dictionary that is in the format that will be uploaded to the server.

Parameters

  • experiment_directory (str) – The experiment{i} directory containing the directory structure as specified in butler.py

  • out_file (str or None) – Absolute path to the output file. “*.json” to write to the output file, None to not write to a file.

Returns

Returns the processed experiment in the form of a dict, which close to the final format for uploading.

Return type

dict

uploading.get_file_names(formatted_dict)

Collects the file names to be uploaded to the server from the formatted dictionary.

Parameters

formatted_dict (dict) – The dictionary as it is output by formatting.experiment_to_json

Returns

A dictionary with:

keys: Space-notation strings to where the files belong on the server. E.g. measurement.png / measurement[“png”] => “measurement png”.

values: Absolute file paths on the local machine.

Return type

dict

uploading.post_measurement(auth_tuple, endpoint='http://127.0.0.1:8000/rest/', dict_path=None)

This posts a measurement to http://endpoint/measurements/

Parameters

  • auth_tuple (tuple[str] or list[str]) – The (user, pass) authentication tuple.

  • uploaded_dicts_statuses (str) – Upload statuses of the dictionaries: true/false.

  • endpoint (str) – Basically the website name. Default is the localhost.

  • dict_path (str) – Path to the formatted dictionary.

uploading.post_measurements(auth_tuple, endpoint, dict_paths, upload_statuses='C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\uploaded_dicts.json', upload_duplicates=False)

Posts posts measurements located in dict_paths and updates the config.uploaded_dicts_json file.

Parameters

  • auth_tuple (tuple[str] or list[str]) – The (user, pass) authentication tuple.

  • endpoint (str) – Basically the website name. Default is the localhost.

  • dict_paths (list[str]) – Paths to the formatted dictionaries.

  • upload_duplicates (bool) – Whether to upload a name that has been flagged as true in config.uploaded_dicts_json

  • upload_statuses (str) – Path to the dictionary with the upload statuses of the paths dict_paths. E.g. “/home/meas_2022_02_02_property.json”: true => already uploaded, don’t upload

utils.dump_numpy_proof(something, fp)

A numpy.array immune json.dump

Parameters

something (list or tuple of np.ndarray or dict or set) – Object to convert to json.

utils.get_experiment_dirs(directory='C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\experiments', rule=None)

Returns all directories in directory that correspond to the format experiment(_.*)x6. E.g. experiment_2022_03_31_21_49_21.

Parameters

directory (str) – The directory where the experiments are lcoated

utils.get_regex(d, pattern, filter_out_nones=True)

Get a value from a dictionary d based on the regex key pattern. NOTE: Takes the shortest matching key.

Parameters

  • d (dict) – The dict.

  • pattern (str) – Any regex. E.g. r”.*property.*” = any string containing “property”

  • filter_out_nones (bool) – Whether to filter out all keys that correspond to None values

utils.get_time_string()

Returns the timedate formatted in %Y_%m_%d_%H_%M_%S

Returns

Formatted timedate in descending order of significance

Return type

str

utils.numpy_to_native(something, inplace=False)

Recursively converts np.ndarrays to lists inside all combinations of nested lists, tuples, dicts, sets

Parameters

  • something (list or tuple or dict or set or np.ndarray) – It has too many np.ndarrays.

  • inplace (bool) – Whether to overwrite the something in-place

Returns

A denumpyified something

Return type

list or tuple or dict or set

utils.replace_json(p, update_dict)

Dump the dictionary update_dict into the file p replacing it.

Parameters

  • p (str) – The path to the file to be replaced.

  • update_dict (dict) – The dictionary to be dumped into p.

utils.update_json(p, update_dict)

Update the JSON dictionary on the path p with the dictionary update_dict.

Parameters

  • p (str) – The path to the file to be updated.

  • update_dict (dict) – The dictionary to be dumped into p.

config.experiment_directory = 'C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\experiments'

The path to the directory containing the experiments.

config.upload_dicts_directory = 'C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\upload_dicts'

The path to the directory containing the formatted experiment JSONs.

config.uploaded_dicts_json = 'C:\\Users\\jhart\\PycharmProjects\\butler\\butler\\uploaded_dicts.json'

The JSON dictionary tracking the upload statuses of experiments in the experiments folder. Warning: Deleting this causes duplicate uploads.

Indices and tables