Backup Universal Analytics Data with Python

| | 4 min read

Universal Analytics End Of Life

Universal Analytics (UA) stopped processing data on July 1st, 2023. Google has announced a complete shutdown of the UA interface and APIs on July 1st, 2024. If you have already transitioned to Google Analytics 4 for your website data collection, this script can assist you in downloading data from your Universal Analytics properties. Downloading data from UA is not as straightforward as you might think. Google imposes a limit of 50,000 API calls per day, and if you have years of data to download with numerous dimensions and metrics, the script can quickly become complex.

Google Universal Analytics Data Archiving

I have created this Python script as an example to any one who is looking to create a script to fetches data using the Google Analytics Reporting API v4. It is designed to retrieve specific metrics and dimensions within a certain date range for a particular view (profile). This script is suitable for simple use cases, such as when you only need data over a short timeline or with a small number of dimensions or metrics.

 

 

Alternate Options: UA Backup with Matomo

Matomo Dashboard

For a more streamlined approach and a UA-like interface to navigate data downloads, consider using Matomo. Matomo is a web analytics solution equipped with a plugin for exporting UA data. Once set up, Matomo handles the backup, allowing you to generate reports as needed through its user-friendly interface.

While there might be some limitations due to the aggregated nature of data downloaded from the Google Analytics API, Matomo captures approximately 90% of the essential data you may want to retain. This includes preserving custom dimensions, eCommerce data, goal data, and more for future reference.

We have successfully assisted many of our customers with downloading UA data to Matomo. If you are considering keeping your Universal Analytics Data, Matomo is an excellent option. We offer UA historical data migration using Matomo as a service. If you are interested, you can checkout a demo here: Matomo Dashboard

Also check

Requirements for the Python Script

  • Python 3
  • googleapiclient
  • oauth2client
  • pyyaml
  • argparse

Setup and Installation

  1. Clone the Repository:
git clone https://github.com/zyxware/ua-archive.git 
cd ua-archive
  1. Install Dependencies: It's recommended to use a virtual environment:
python3 -m venv venv source venv/bin/activate 
pip3 install google-api-python-client oauth2client argparse pyyaml
  1. Service Account and API Key:
  • Set up a service account in Google Cloud Console. - Follow Google Documentation for the detailed instructions.
  • Download the JSON key file.
  • Rename the file to ua-archive.json and place it in the project directory.
  1. Configure the Script:
    • Copy 'settings.yml.defualt' to 'settings.yml' and update 'api_key' with the name of json file generated and 'view_id' with your UA property's view ID.
    • Edit the reports_config.yml to add the reports you want to generate.

 

 

Usage

Run the script with Python:

python3 analytics_reporter.py --report_id 1 --start 2023-01-01 --end 2023-01-31

After successful execution, a CSV file named with the corresponding report name will be generated in the project directory.

Default Reports

The system supports the reports defined in reports_config.yml - You can modify the reports_conf.yml to add the dimensions and matrices you want to include in the report you want to generate. The full set of available dimensions and metrics are available on Google UA Dimensions & Metrics Explorer

reports:
  - id: 1
    name: "User Engagement Over Time"
    dimensions: ["ga:clientId", "ga:date"]
    metrics: ["ga:sessions", "ga:sessionDuration", "ga:pageviews"]

  - id: 2
    name: "User Journey Analysis"
    dimensions: ["ga:landingPagePath", "ga:exitPagePath", "ga:pagePath", "ga:previousPagePath"]
    metrics: ["ga:sessions", "ga:pageviews", "ga:entrances", "ga:exits"]

  - id: 3
    name: "Acquisition Channels and Campaign Performance"
    dimensions: ["ga:source", "ga:medium", "ga:campaign"]
    metrics: ["ga:sessions", "ga:users"]

  - id: 4
    name: "User Segmentation"
    dimensions: ["ga:userAgeBracket", "ga:userGender", "ga:userType"]
    metrics: ["ga:users", "ga:sessions", "ga:sessionDuration", "ga:pageviews"]

  - id: 5
    name: "Conversion Analysis"
    dimensions: ["ga:goalCompletionLocation", "ga:goalPreviousStep1", "ga:goalPreviousStep2", "ga:goalPreviousStep3"]
    metrics: ["ga:goal1Completions", "ga:goal1ConversionRate"]

  - id: 6
    name: "User Interaction with Content"
    dimensions: ["ga:pagePath", "ga:title"]
    metrics: ["ga:pageviews", "ga:uniquePageviews", "ga:avgTimeOnPage", "ga:entrances"]


  - id: 7
    name: "External Keyword and SEO Analysis"
    dimensions: ["ga:sourceMedium", "ga:landingPagePath"]
    metrics: ["ga:sessions", "ga:users", "ga:newUsers", "ga:pageviews", "ga:bounceRate", "ga:sessionDuration"]
    
  - id: 8
    name: "Internal Site Search Analysis"
    dimensions: ["ga:searchKeyword", "ga:searchStartPage", "ga:searchCategory"] 
    metrics: ["ga:sessions", "ga:users", "ga:pageviews", "ga:searchResultViews", "ga:searchUniques"]


  - id: 9
    name: "Device and Technology Analysis"
    dimensions: ["ga:deviceCategory", "ga:browser", "ga:operatingSystem"]
    metrics: ["ga:sessions", "ga:users", "ga:pageviews"]

  - id: 10
    name: "Custom User Events Tracking"
    dimensions: ["ga:eventCategory", "ga:eventAction", "ga:eventLabel"]
    metrics: ["ga:totalEvents", "ga:uniqueEvents"]

Do let us know if you need professional support in backing up your UA data. We can help you in setting up Matomo on your own infrastructure and configure it to create an archive of your Universal Analytics data.

Sample Report: Channel Grouping

UA Channel Grouping

Channel Grouping is one of the most popular reports our clients are interested in downloading. To illustrate how the Channel Grouping report will appear after being backed up, we have created a demo report in Looker Studio. This demo provides a clear view of what to expect from the backup process.

To create a channel grouping report you can use the following configuration:

reports:
  - id: 1
    name: "Channel Grouping"
    dimensions: ["ga:date", "ga:channelGrouping"]
    metrics: ["ga:users", "ga:newUsers", "ga:sessions", "ga:bounceRate", ga:pageviewsPerSession, ga:avgSessionDuration]

Sample Report: Annotations in UA

UA Annotations

Annotations are another feature from Universal Analytics (UA) that many users want to download for reference. This feature is extensively used to mark major marketing events or changes happening on their websites. To assist in visualizing how annotations will be preserved, we have prepared a demo in Looker Studio. This demo showcases how the annotations will appear after being backed up.