AWF docs

# Getting started

AWF provides a Python SDK to simplify chains of requests and networking to a few lines of code. You can get all the information available regarding Workflows, Runs, Resources and many more in one dependency.

# 🔨 Installation

Use the package manager pip (opens new window) to install connections.

pip install git+https://gitlab.arup.com/awf/awfpy.git@v2.4.0

# ⚙️ Objects

AWF currently has a set of object types that it uses to function, some of them being created by users, the rest being managed by the system. Resources and Runs can be managed and manipulated at will by the user. This allows you to programmatically run any Workflow published in any of the three environments. The rest of the objects, such as Workers, Workflows, Modules, etc. are read-only.

# 🎓 Run the tutorial

Everything revolves around the AWFClient. The methods and AWF objects are all documented and typed. Here's an example using the SDK to create and start a tutorial workflow.

# 🔗 Dependencies

We need to import everything we need from the SDK. We will import the client that connects to the API and contains the methods required for us. We will also import the sleep function for one of the examples below.

from awfpy import AWFClient
from awfpy.objects import Run, Resource, ResourceType, Environment, Status
from time import sleep

# 🧰 Client

The AWF client connects to the AWF API to communicate with AWF. There are several ways to instantiate the client, but the simplest form is shown below:

client = AWFClient(url="https://awf.arup.com/api/")

In this case where no argument is supplied besides the url, the client will prompt you in a new browser window to login with your personal account. You will be given a code that is automatically added to your clipboard, you can Ctrl+V to paste the code in the browser window and continue the authentication as you would usually do.

💡 TIP FOR AUTOMATION

If you are using awfpy in a fully automated process, it may not be feasible to have a user sign in to authenticate. For these scenarios you can reach out to the AWF development team to get a dedicated API Key and then add it to the client:

client = AWFClient(url="https://awf.arup.com/api/", api_key="your_api_key")

# 👨‍🏭 Workflow selection

Get the workflow that we are planning to use, in our case Tutorial. We can either use the slug (shorthand name, all lowercase, not easy to find), or use the label, which we know from the AWF website. In the example above we didn't know what's the slug for sprayed-concrete-lining, and we used the full label.

tutorial_workflow = client.get_workflow('tutorial')

# If you don't know the shorthand of a workflow, you can use the label
cpt_workflow = client.get_workflow_by_label('Sprayed Concrete Lining Optimizer')

# 🕵️ Workflow arguments

Workflows usually require inputs which are up to you to choose. On the website you can interactively select them, here we need to check what's their ID and what sort of information they need. For example, here's the tutorial workflow arguments.

The most important fields in an argument are name, which we need to remember to give it back when creating a Run, the type, so we know what we can write to it, and label or description, so we know what it actually is if it's not recognizable from the start.

We can check the arguments below, and we see that object_id is actually just the Run name, and you can name it anything you'd like! Other than that, the input file needs to be a file, and we need to supply those using resources, ignore that for now, we'll handle it below.

print(tutorial_workflow.arguments)

[
  {
    "demo": "integration_test",
    "name": "object_id",
    "type": "string",
    "label": "Workflow Run Name",
    "default": "",
    "required": true,
    "description": "Add a name to your workflow"
  },
  {
    "demo": "demo-files/tutorial/demo.txt",
    "name": "file",
    "type": "file",
    "label": "Input file",
    "default": "",
    "required": true,
    "properties": {
      "extension": [".txt"]
    },
    "description": "A text file to run tutorial workflow on"
  }
]

# 🚀 Run creation

Create the Run object, containing the workflow from earlier, owner and arguments. A Run can contain so much more information, be sure to check that out, but these are the required parameters. And you can see we are wrapping this Run object that we are creating in client.create_run, which actually sends the Run to the API to be created. If we take that out, the Run would not be created on the server-side, and it wouldn't do anything.

🙌 IMPORTANT

Please replace JOB_NUMBER with your own job number. It's used internally for usage monitoring, cost distribution and much more.

run = client.create_run(Run(
    workflow=tutorial_workflow.url,
    owner='awfpy-test@awf.com',
    job_number=JOB_NUMBER,
    arguments={
        "object_id": "My own tutorial run!"
    }
))

# 🗄️ Resource creation

Similarly, we create a Resource Object, since the Tutorial expects a file as an input. There are a few more parameters required this time around. You can find more documentation about each and every parameter, including optional ones, in the code documentation. In short, we are giving it the basic information needed like name and description and the type of file it is. The magic to resources is related to filename, that indicates the actual file on disk that we want to upload to AWF to be used in a specific Workflow.

resource = client.create_resource(Resource(
    name='tutorial.txt',
    description='Input file',
    filename='files/tutorial.txt',
    type=ResourceType.INPUT,
    activity="UI",
    run=run.url
))

# ⚡ Run submission

And this is our final step, submitting the run, marking it as ready to be picked up and executed by AWF Workers.

Note: be sure to create all the resources and give all the inputs you need for your workflow run before you submit it, otherwise it might fail!

client.submit_run(run)

# ⏲️ Getting status updates

All client methods that take AWF objects as parameters are updated. Therefore, we can use methods like check_run_status and our run object will be up-to-date. We can run a check in a while loop to check for it to be done!

# You could check for other statuses like unsuccessful statuses
# You can check for a collection of statuses using the integer value of the enum
#   >>> run.status.value > Status.STARTED.value

while run.status.isRunning():
    client.check_run_status(run)
    sleep(3) # Wait three seconds

# 📜 Gettings real-time logs

In the while loop above we could also add some print method to get the logs, but the output will be cluttered with almost the same logs with some new information appended.

Most of the time AWF users just want to run a workflow, and wait for it to finish while looking at logs to make sure everything is going as expected. For use cases like these we have a built in function to stream the logs to the console.

While streaming logs, it will block the current thread, which means that until the workflow is either successfully completed or failed somewhere during execution, it will keep on printing new logs without repeating old information.

client.stream_logs(run)

# 📁 Download resulting files

Using download_resource you can download the files locally. There are optional parameters as well if you prefer to download them in a specific folder or path, or if you want them to stay zipped.

for resource in run.resources:
    if resource.type == ResourceType.OUTPUT:
        client.download_resource(resource)

# 🗂️ Download all files

You can also choose to download all the resources associated with a Run. It will download all input, output and logs related to the run.

client.download_resources_from_run(run)

← Unit Testing →