# External Notebooks source: https://docs.chalk.ai/docs/notebook-development ## Use Chalk from the external notebook of your choice. You can write and test new features from an external notebook: Jupyter, Colab, VS Code, or a hosted environment like Deepnote or Hex. To get started, you'll want to install and connect Chalk to your external notebook environment. Generally this involves two steps: - Installing the chalkpy[runtime] package, - Authenticating to Chalk (using the chalk CLI or by setting environment variables) We provide specific setup instructions for Deepnote, Hex and Vertex, but the process should be similar for other notebook environments. ### Installing the Chalk Python Dependency To install dependencies, run the !pip install "chalkpy[runtime]" bash command directly in a cell. Alternatively, you can add and install the chalkpy[runtime] dependency in your kernel's virtual environment. ### Connecting Your External Notebook to Chalk Authenticating takes a client ID and client secret pair. Authentication for External Notebooks covers generating that pair and wiring it into a local notebook, Hex, Deepnote, or Vertex. Once the credentials are available to your notebook, construct a client: ``` import os from chalk.client import ChalkClient client = ChalkClient( client_id=os.environ.get("CHALK_CLIENT_ID"), client_secret=os.environ.get("CHALK_CLIENT_SECRET"), ) ``` ### Verifying the Setup To verify that everything is properly set up, you can run the following code in a cell: ``` client.whoami() ``` If everything is properly configured, you should see your user_id, team_id and environment_id come back: ``` WhoAmIResponse(user='', environment_id=, team_id='') ``` ### Loading Features Now we can start building some new features! To load your Chalk features into your notebook, you'll want to run: ``` from chalk.client import ChalkClient client = ChalkClient() client.load_features() ``` This will print out your available features and load them into your notebook: ``` Director ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Feature ┃ Type ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ id │ str │ │ name │ str │ │ birth_year │ int │ │ active │ bool │ │ favorite_numbers │ list[int] │ │ death_year │ int | None │ │ known_for_titles │ list[str] │ │ known_for_genres │ list[str] │ │ movies │ DataFrame[Movie] │ │ num_movies │ int │ └────────────────────┴───────────────────────┘ Movie ┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Feature ┃ Type ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ id │ str │ │ director_id │ str │ │ release_number │ int │ │ type_ │ str │ │ title │ str │ │ original_title │ str │ │ release_date │ dt.datetime │ │ runtime_m │ float │ │ primary_genre │ str │ │ secondary_genre │ str │ └────────────────────────┴───────────────────┘ ``` We'll use the data schema above for the rest of this tutorial. Before we get started writing features, you'll want to import _ from Chalk. _ (underscore) allows you to build out features on your feature classes. ``` from chalk.features import _ ``` Additionally, if you want to create and work from a new branch (which is recommended), run the following command: ``` client.get_or_create_branch(branch_name="") ``` This command will: - create a new Chalk branch deployment that has the same features defined as your main Chalk deployment, - point your client to that new branch. If the branch already exists, your client will just be updated to point to that branch (no new branch deployment will be created). You can also point a client at a branch by passing branch= to the ChalkClient constructor. That form requires the branch to exist already, so prefer get_or_create_branch when you are not certain it does. ### Building New Features Let's say we want to write a new "exclaimed name" feature, which is a director's name with an exclamation mark tacked on. To do this, run the following code in a cell: ``` Director.name_exclaimed: str = _.name + "!" ``` In this example, the underscore acts as a reference to the Director feature class. We have now created a new feature in our notebook and can query for it in either an online or offline context. ### Querying for features To query for your new features, pass them as outputs to the ChalkClient query methods. For instance, to run an online query for your new Director.name_exclaimed feature, you can run the following: ``` new_features = client.query( input={ Director.id: 113, }, output={ Director.name, Director.name_exclaimed, } ) new_features ``` Running an offline query is equally straightforward: ``` chalk_dataset = client.offline_query( input={ Director.id: list(range(1000)), }, output={ Director.id, Director.name, Director.name_exclaimed, }, recompute_features=True, run_asynchronously=True, dataset_name="", ) df = chalk_dataset.to_pandas() ``` Running the offline query above creates a Pandas DataFrame with the columns Director.id, Director.name, and Director.name_exclaimed (alongside some metadata columns). This DataFrame will have one row for every Directors with an ID between 0 and 999. By providing a dataset_name to the offline_query, we've created a named dataset. Named datasets can easily be shared with other team members. Any historical dataset can be accessed using the ChalkClient. Named datasets can be accessed as follows: ``` client = ChalkClient() dataset = client.get_dataset(name="") ``` ### Building Out More Complex Features You can use expressions to build out more complex features. Generally these more complex features are composed of: has-one relationships, has-many relationships, and chalk functions. ### Using Has-one Relationships In the schema we've defined above, a Movie is in a has-one relationship with its Director (even if this is not necessarily true in practice). When building new features on the Movie class, features from joined classes can be directly accessed. For instance, let's say we want to define Movie.is_latest_release. We can accomplish this by defining a feature that reaches through the has_one join: ``` Movie.is_latest_release: bool = _.director.num_movies == _.release_number ``` The expression above defines a boolean feature which tests whether the release number for a movie (_.release_number) is equal to the number of movies a director has released (_.director.num_movies). ### Using Has-Many Relationships In the schema we've defined above, a Director is in a has-many relationship with their Movies. When building new features on the Director class, features from joined classes can be directly accessed. For instance, let's say we want to define Director.average_movie_runtime. We can accomplish this by defining an aggregation feature: ``` Director.average_movie_runtime: float = _.movies[_.runtime].mean() ``` In the above example, the two different _'s refer to two different feature class namespaces: - _.movies can be thought of as Director.movies, - _.runtime can be thought of as Movie.runtime. The expression above gets a directors movies (_.movies), specifies that the aggregation is happening on the runtime field of those movies (_.movies[_.runtime]) and then calculates the mean (.mean()). Filters can also be passed to these expressions. For instance, say we want to filter out any movie that has a primary or secondary genre of short from the average runtime. This can be done as follows: ``` Director.average_movie_runtime_filtered: float = _.movies[ _.runtime, # aggregation column _.primary_genre != "short", # filter _.secondary_genre != "short" # additional filter ].mean() ``` ### Using Chalk Functions Chalk exposes a number of functions that can be called and composed as part of expressions. The expressions docs list all of them. Below, we provide a few examples of how these functions can be used to create new features. Import chalk.functions once, and it's available to every cell in the session: ``` import chalk.functions as F ``` Identify whether a movie belongs to a Director's typical genres: ``` Movie.is_directors_usual_genre: bool = F.contains(_.director.genres, _.primary_genre) ``` Identify whether a movie was released on a US federal holiday: ``` Movie.released_on_holiday: bool = F.is_us_federal_holiday(_.release_date) ``` Get release_date of Director's last movie: ``` from datetime import datetime Director.latest_movie_release: datetime = F.max_by(_.movies[_.release_date], _.release_date) ``` ### Composing New Features The features you have defined in your notebook can also be composed into new features. For instance, you can define the Director.makes_movies_that_are_too_long feature: ``` Director.makes_movies_that_are_too_long: bool = _.average_movie_runtime > 150 ``` ### Translating this Code into Production Code Defining features as expressions has two advantages: - Features stay small and granular, so you can compose them into larger ones. - Everything written in this tutorial translates directly to your Chalk feature store, with no rewrite when you move it into your project.