Python basics

Setup

Everything needs to be installed before getting to this stage. The examples are using a conda env.

pandas

import numpy as np
import pandas as pd

Polars

import polars as pl

DuckDB

import duckdb

Arrow

import pyarrow as pa

Creating

pandas

It may be easiest to start from a dictionary.

# create the column dictionary
input_dict = {
    'languages': ['Fortran', 'R', 'Python', 'Julia'],
    'created': [1957, 1993, 1991, 2012],
    'has_syllabus': [False, True, True, True]
    }
input_dict

{'languages': ['Fortran', 'R', 'Python', 'Julia'],
 'created': [1957, 1993, 1991, 2012],
 'has_syllabus': [False, True, True, True]}

Convert to DataFrame, with keys becoming the column headers:

df = pd.DataFrame(input_dict)
df

	languages	created	has_syllabus
0	Fortran	1957	False
1	R	1993	True
2	Python	1991	True
3	Julia	2012	True

The default display does not show types, so get these in an extra step.

df.dtypes

languages         str
created         int64
has_syllabus     bool
dtype: object

Polars

# create the column dictionary
input_dict = {
    'languages': ['Fortran', 'R', 'Python', 'Julia'],
    'created': [1957, 1993, 1991, 2012],
    'has_syllabus': [False, True, True, True]
    }
input_dict

{'languages': ['Fortran', 'R', 'Python', 'Julia'],
 'created': [1957, 1993, 1991, 2012],
 'has_syllabus': [False, True, True, True]}

Convert to DataFrame, with keys becoming the column headers:

pl_df = pl.DataFrame(input_dict)
pl_df

shape: (4, 3)

languages	created	has_syllabus
str	i64	bool
"Fortran"	1957	false
"R"	1993	true
"Python"	1991	true
"Julia"	2012	true

Note that the default display is not really Pythonic, with double-quoted string and lowercase bools. We are working with a thin layer over Rust.

DuckDB

Creating a data source from scratch within DuckDB is relatively unusual.

Just use whatever you have: pandas or Polars dataframes, CSV, Parquet or JSON files, remote database connections…

Arrow

TODO

Importing data

pandas

Polars

TODO

DuckDB

TODO

Arrow

TODO

Subsetting

An important difference is whether row names are used (pandas) or not (Polars, SQL).

pandas

In our toy example, we set column names:

df.columns

Index(['languages', 'created', 'has_syllabus'], dtype='str')

Row “names” are called the index in pandas. We did not set them explicitly, so pandas falls back to the implicit index of sequential integers.

The explicit/implicit distinction is important in pandas indexing, and in Python “explicit is better than implcit”.

df.index

RangeIndex(start=0, stop=4, step=1)

To get a cell value, the .at attribute uses explicit indices, .iat uses implicit. The [row, col] syntax copies Numpy.

df.at[2, 'languages']

'Python'

df.iat[1, 0]

'R'

Retrieve a single column with indexing, or use the column name as an attribute.

df['languages']

0    Fortran
1          R
2     Python
3      Julia
Name: languages, dtype: str

df.languages

0    Fortran
1          R
2     Python
3      Julia
Name: languages, dtype: str

For multiple columns, use .loc or .iloc with a list or a range of columns. As in NumPy, the :, means all rows:

df.loc[:, 'languages':'created']

	languages	created
0	Fortran	1957
1	R	1993
2	Python	1991
3	Julia	2012

Note that pandas ranges differ from core Python, as both the start and the stop are included.

For regex matching of column names, use the filter() method (which can also do other things).

df.filter(regex = '^[lh]')

	languages	has_syllabus
0	Fortran	False
1	R	True
2	Python	True
3	Julia	True

Polars

TODO

DuckDB

TODO

Arrow

TODO

Sorting

pandas

Polars

TODO

DuckDB

TODO

Arrow

TODO

Modifying

pandas

Polars

TODO

DuckDB

TODO

Arrow

TODO