# Datawisp Documentation Welcome to Datawisp's official documentation site. Here you'll find more detailed information on what Datawisp is, how it works, how to use it, and links to other useful resources. Datawisp is evolving quickly as we try to create the best possible experience for our users. We're constantly expanding this document and making it more comprehensive, however screenshots / references to the product may become outdated from time to time. Don't hesitate to contact us directly with any questions at . # Datawisp Overview Datawisp is a no code analytics platform that lets anyone analyze data and discover insights regardless of their technical knowledge. Its node-based interface allows users to combine different “blocks” to transform data, much like a developer would with code. Each block performs a different operation, such as filtering, sorting, or creating a chart, and linking even 3 or 4 of these blocks can transform and analyze data in powerful ways. Datawisp also features an AI-powered assistant, named Wispy, that takes natural language input from the user and automatically creates charts, tables, and more. Worksheets created by Wispy can be used as-is or can be subsequently edited manually by anyone with some knowledge of how blocks work. Datawisp runs in a browser and lets users collaborate in real time and share their work with both internal and external audiences. # Adding your data Datawisp can * **Connect to your existing infrastructure** (databases, data warehouses, data lakes, ...)\ If you already have your data stored in some form of database, this is the recommended approach. Data is processed on your infrastructure. The full dataset is never copied.
* **Work with files uploaded to Datawisp** (Excel, CSV, JSON, ...)\ Alternatively, you can upload documents to Datawisp. In that case, the Datawisp servers will store and process the data. ### Supported data sources #### **Data warehouse platforms** * Snowflake * Google BigQuery * AWS Redshift * MotherDuck * PlanetScale #### **Databases** * MariaDB * MySQL * MariaDB * Microsoft SQL Server * PostgreSQL **Note:** databases are supported regardless of your provider. Datawisp supports EC2, Google Cloud databases, Azure Database, PlanetScale, ...) #### Files * Excel (xlsx) * JSON * CSV * Google Sheets **Note:** to import a Google Sheet, link sharing must be enabled ("Anyone with the link can see"). #### Other Applications, CRMS, Analytics Tools, ... Datawisp supports other tools, such as: * Google Analytics * Firebase * Honeycomb * AppsFlyer * ... Many applications like those allow Datawisp to connect directly to them via a data platform (e.g. Google Analytics / Firebase data can be accessed via BigQuery) with a little setup. In most other cases, our onboarding team will help you connect the data to Datawisp. ### Overview Typically, our team will help you add your data to Datawisp. This is included in every Datawisp subscription, including free trials. Data is always added to Datawisp via the [data page](https://app.datawisp.io/my-datasources), which you can access on the left sidebar.

The Data Page

To add data, you can use the "**Add Data**" button. For some data sources, you can find additional help on one of the subpages. And if you have any trouble, please don't hesitate to ask. ### Data Dictionaries After adding your data to Datawisp, we recommend [adding a Data Dictionary](/adding-your-data/data-dictionaries). ### Security Datawisp always connects to your database via an encrypted connection. #### TLS Certificates Many cloud providers unfortunately do not deploy valid certificates for their databases. In this case, Datawisp will offer you to pin the certificate it receives at import time. After a certificate is pinned, Datawisp will refuse to connect if your database sends a different certificate. #### Whitelisting To allow Datawisp to connect to your database, you may need to whitelist Datawisp's IP in the respective firewall. The current IP is always shown in the add data view.

Add Data View

Please refer to the documentation of your firewall if Datawisp is unable to connect to your database. If you still need help, don't hesitate to reach out to our team. #### Private Networks, NATs, Gateways, VPNs, Private Clouds, ... If your setup requires additional authentication to connect to the database, please reach out to our team. # Connect to a database # Snowflake This guide will walk you through connecting Datawisp to your Snowflake databases. We assume that you have an existing Snowflake account which is already storing some data and that you are familiar with the basics of using Snowflake. {% hint style="info" %} This guide requires that you have at least `USERADMIN` permissions on your Snowflake account. If you do not have these permissions, or aren't sure, e.g. because someone else is administering the account, please send this guide to them. {% endhint %} To get started, log into your Snowflake account. Create a new SQL worksheet, then paste and run the following commands: ```sql -- create a role you can use to grant privileges to Datawisp CREATE ROLE IF NOT EXISTS datawisp; -- OPTIONAL: create a dedicated warehouse for Datawisp -- the settings below are our recommendations, you may tune them to your needs CREATE OR REPLACE WAREHOUSE datawisp_wh WAREHOUSE_SIZE = SMALL -- appropriate for small datasets, increase as needed AUTO_SUSPEND = 120 AUTO_RESUME = TRUE INITIALLY_SUSPENDED = TRUE ; -- create a user for Datawisp to log in with CREATE OR REPLACE USER datawisp TYPE = SERVICE DEFAULT_WAREHOUSE = 'DATAWISP_WH' -- change if you skipped the last step DEFAULT_ROLE = 'DATAWISP' ; GRANT ROLE datawisp TO USER datawisp; -- change warehouse name if you're using an existing warehouse GRANT USAGE ON WAREHOUSE datawisp_wh TO ROLE datawisp; ``` After successfully running these commands, grant permissions on the database you want to import to the newly created `datawisp` role. You can either do this manually through the Snowflake UI or using SQL commands: ```sql SET mydb = 'MY_DATABASE'; -- replace with the name of the database to grant access to GRANT USAGE ON DATABASE IDENTIFIER($mydb) TO ROLE datawisp; GRANT USAGE ON ALL SCHEMAS IN DATABASE IDENTIFIER($mydb) TO ROLE datawisp; GRANT USAGE ON FUTURE SCHEMAS IN DATABASE IDENTIFIER($mydb) TO ROLE datawisp; GRANT SELECT ON ALL TABLES IN DATABASE IDENTIFIER($mydb) TO ROLE datawisp; GRANT SELECT ON FUTURE TABLES IN DATABASE IDENTIFIER($mydb) TO ROLE datawisp; ``` Now open a new browser tab or window and go to Datawisp. We are going to switch back and forth between Snowflake an Datawisp a few times. Head over to the Data Sources page and press **Import**.
Select **Database** and change the *Connection type* to **Snowflake**.
Choose a *Title* for the datasource. Then go back to Snowflake. Click on the button in the bottom left, then hover over *Account*. This pops up a second menu with all your Snowflake accounts. There, hover over the account you're logged in with (the one with the Checkmark). This pops up a third menu. There, in the top right corner, click the **Copy account identifier** button. Go back to Datawisp and paste this under *Account Identifier*.
Under *Database*, type the name of the database in Snowflake that you wish to import. {% hint style="warning" %} Carefully verify that the database name is capitalized exactly as in Snowflake. Database names in Snowflake are **UPPERCASE** by default, so this is how you should enter them in Datawisp. Otherwise, you may later run into errors *even if the connection initially succeeds and appears to work*. {% endhint %} Under *Username*, type `datawisp`.
Then click on the **ALTER USER** command under *RSA Public Key*. Head back to Snowflake and run this command. {% hint style="info" %} For security reasons, Datawisp always creates a new RSA key when you import a Snowflake database. Until you complete the import, this key does not leave your PC. This may cause problems in certain edge cases such as the following: * You send this guide to a colleague because you don't have `USERADMIN` privileges on Snowflake. Your colleague runs through all the steps, including the **ALTER USER** command at the end that sets the RSA key. Then your colleague sends the connection details to you. You fill in the data and press **Connect**. The connection fails because Datawisp attempts to connect with your key, but on Snowflake your colleague's key was registered. {% endhint %} Then go back to Datawisp and click **Connect**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b11.png) If everything worked, Datawisp will tell you that the connection to your Snowflake database was successful: ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b12.png) # BigQuery This guide will walk you through connecting Datawisp to your existing BigQuery datasets. We assume that you have an existing account on Google Cloud Platform (GCP) which is already storing data in BigQuery. {% hint style="info" %} If you are importing a BigQuery dataset created by Google Analytics, please follow the GA-specific instructions: [Google Analytics via BigQuery](/adding-your-data/connect-to-a-database/google-analytics-via-bigquery) {% endhint %} ## GCP Setup We will begin by setting everything up on the GCP side. This is done using the [Google Cloud Console](https://console.cloud.google.com/). Before we begin, make sure that the right **project** is selected in the top left corner. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b01.png) ### Enable BigQuery API In the search box at the top of the page, search for and select *BigQuery API* (or click [here](https://console.cloud.google.com/marketplace/product/google/bigquery.googleapis.com)) and make sure that it says *API Enabled* - otherwise click the "Enable" button ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b00.png) ### Create a Service Account for Datawisp In the search box at the top of the page, search for and select *Service Accounts* (or click [here](https://console.cloud.google.com/iam-admin/serviceaccounts)). ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b02.png) Then click **Create Service Account**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b03.png) Pick an account name, ID, and description. Make sure to copy the shown *Email address* and save it somewhere. You're going to need it later. Then press **Create and Continue**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b04.png) Select the **BigQuery Job User** role. This will allow Datawisp to run queries in your BigQuery account. Press **Done**. ### Create a key to the Service Account for Datawisp to log in with In the list, select the service account you just created. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b08.png) Go to **Keys**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b09.png) Click **Create new key**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b0a.png) Keep **JSON** selected and click **Create**. Note that your browser will now download a file. This file contains all the information required to log into the service account. You will need this later, so you can give Datawisp access to the service account. ### Grant Access to the Dataset to the Service Account {% hint style="info" %} If you are importing Google Analytics data, [this step needs to be done differently](/adding-your-data/connect-to-a-database/google-analytics-via-bigquery#grant-access-permissions). {% endhint %} Go to [BigQuery](https://console.cloud.google.com/bigquery). Select the dataset you want to import to Datawisp. Copy the dataset name and save it somewhere. You are going to need it later.
Then click **Sharing**->**Permissions**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b06.png) Click **Add Principal**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b07.png) Under **New principals**, paste the service account *Email address* that you saved previously. Assign the **BigQuery Data Viewer** role. This will allow Datawisp to see all tables and views in the dataset. Alternatively, you may choose to grant this role only on specific tables inside the dataset. In that case, you still need to grant the **BigQuery Metadata Viewer** role on the on the dataset. This will allow Datawisp to list (but not query) all tables in the dataset and is needed so that Datawisp can find the tables that it was granted access to. Then click **Save**. ## Datawisp Setup Head over to the Data Sources page and press **Import**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b0d.png) Select **Database** and change the *Connection type* to **Google BigQuery**.
Choose a *Title* for the datasource. For *Service Account Credentials* select the JSON file with the keys that you downloaded previously. Under *Dataset* paste the dataset name that you copied previously. In our example, it looks like this:
Then press **Connect**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b11.png) If everything worked, Datawisp will tell you that the connection to your BigQuery database was successful: ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b12.png) # Google Analytics via BigQuery When connecting Google Analytics (GA) to BigQuery, your data is stored in a format known as *sharding*. Table sharding is the practice of storing data in multiple tables, using a naming prefix such as `[PREFIX]_YYYYMMDD`. This presents a challenge when doing analytics on the data, as the data is not in a single table (the BigQuery documentation also [recommends against using table sharding](https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard)). When importing this directly into Datawisp, Datawisp would treat each shard as a separate table -- this would lead to a very poor experience overall as you end up with hundreds of tables when you logically only really want one. In order to solve this, we have created this guide to show you how to create a view in BigQuery that will present the data in a single table to Datawisp. ## Create a view in BigQuery First, we need to create a view in BigQuery that will present the data in a single table to Datawisp. Open the BigQuery console create a new SQL query. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b15.png) There, paste the following SQL statements: ```sql CREATE OR REPLACE VIEW `.ga_events` AS SELECT * EXCEPT(event_params), parse_json((SELECT CONCAT( '{', STRING_AGG( CONCAT( TO_JSON_STRING(ep.key), ': ', COALESCE( NULLIF(TO_JSON_STRING(ep.value.int_value), 'null'), NULLIF(TO_JSON_STRING(ep.value.string_value), 'null'), NULLIF(TO_JSON_STRING(ep.value.double_value), 'null'), NULLIF(TO_JSON_STRING(ep.value.float_value), 'null'), 'null' ) ) ), '}' ) FROM UNNEST(event_params) AS ep ), wide_number_mode=>'round') as event_params FROM `.events_*`; CREATE OR REPLACE VIEW `.ga_users` AS select * from `.users_*`; CREATE OR REPLACE VIEW `.ga_pseudonymous_users` AS select * from `.pseudonymous_users_*`; ``` {% hint style="info" %} Google Analytics supports exporting both **Event Data** (`events_YYYYMMDD`) and **User Data** (`users_YYYYMMDD`, `pseudonymous_users_YYYYMMDD`). This guide shows you how to import *all* of those tables into Datawisp. If you don't have some of them (e.g. because you did not enable the export of User Data), simply omit the respective steps. For example, if you don't have any `events_YYYYMMDD` tables, simply remove the first `CREATE VIEW` statement in the above SQL snippet. {% endhint %} ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b17.png) **Make sure to replace `` with the name of your dataset.** Then press **Run**. ## Grant Access Permissions At this point the GA-specific preparations are complete, so you can now get started with the generic [BigQuery import guide](/adding-your-data/connect-to-a-database/bigquery). Follow this guide until you reach the step [Grant Access to the Dataset to the Service Account](/adding-your-data/connect-to-a-database/bigquery#grant-access-to-the-dataset-to-the-service-account). At this point, it is important that you do **not** grant the *BigQuery Data Viewer* role on the entire dataset to the service account. Grant the *BigQuery Metadata Viewer* role instead. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b18.png) Then, on the dataset go to **Sharing** -> **Authorize Views**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b19.png) Authorize the views that you created in the previous step. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b1d.png) Then, on each of the views, grant the *BigQuery Data Viewer* role. This setup effectively hides all the sharded tables from Datawisp and only grants access to the views. Then, proceed with the [rest](/adding-your-data/connect-to-a-database/bigquery#datawisp-setup) of the guide. Datawisp will now be able to import your GA data correctly. # MotherDuck This guide will walk you through connecting Datawisp to your existing MotherDuck database. To get started, head over to and click **+ Create token**.
The *Token type* should be set to **Read Scaling Token** as Datawisp will never write to your database. After clicking **Create token**, copy the generated token and head back to [Datawisp](https://app.datawisp.io/my-datasources). Head over to the Data Sources page and press **Import**.
Select **Database** and change the *Connection type* to **MotherDuck**.
Choose a *Title* for the datasource. Under *Token,* paste the access token you just created. For *Database* enter the name of the MotherDuck database database that you want to connect to. Then click **Connect**. ![](https://md.statshelix.com/uploads/a1fd0fba00c538a8090828b11.png) If everything worked, Datawisp will tell you that the connection to your MotherDuck database was successful:
# Tips for Success Coming soon. # Troubleshooting Data imports must be properly formatted for Datawisp to load them correctly. A few common pitfalls are: 1. Incorrect Position of Data 2. Incorrect Headers 3. Uncommon (or missing) Separators 4. Extraneous Information ## 1. Incorrect Position of Data When opened in a spreadsheet (google sheets, excel, etc) properly formatted csv documents start in cell A1, the upper left hand corner. Correct:
Incorrect:
To remedy these issues, you can delete the empty rows and columns, or select and copy your data into a new spreadsheet starting at A1. ## 2. Incorrect Headers Datawisp needs column headers to use as column names. This can be seen in a spreadsheet, or by opening your document in notepad or similar. Correct:
Notice that regardless of the column type (numeric, string, etc) the first full row are strings describing the data you see in the rest of the column. incorrect:
Compare to here where the first row is the dataset, without a header column. A good indicator of this is if your numeric columns seem to have numeric entries in row 1. Column headers are rarely numeric. ## 3. Uncommon (or missing) Separators In contrast to diagnosing the first two issues, this is easier to diagnose in notepad or some other text editor. Open your csv in one of these editors and ctrl+f to find the character "," (a comma). There should be one separating every "cell" of data. Once you see you have a comma or a tab separating your data, you can open in a spreadsheet to be certain the number of rows and columns are consistent throughout the data, which is usually a symptom of an incorrect separator correct:
Notice how a comma separates each column of the data, multiple consecutive columns indicate missing data. incorrect:
Notice here there are no clear separators for columns. While it might appear to be spaces separating columns, note above some entries like "Ground Plane" are a single entry, whereas without comas it would be unclear. ## 4. Extraneous Information In spreadsheet tools, it's not uncommon to have a dataset, and then do calculations in the same sheet over to the side or below the dataset. This creates many issues for a csv. When saving a spreadsheet as a csv, ensure there is only a single dataset in the sheet, and that the datset is complete. Having extra text or calculations in other places on the sheet could cause unexpected behaviors throughout the document. # Reloading the database schema ### When to reload the database schema Datawisp only connects to your database when the user uses the data. Therefore Datawisp is not automatically informed when you create, change or delete tables or views. This means that sometimes the schema of your database, and the schema that Wispy or you see is different. To resolve this, it is recommended that you reload the schema after: * You create or delete a table or a view * You alter the schema of a table or a view by adding or removing columns, or changing their type * You change the permissions for tables or views that are imported into Datawisp. ### Updating the database schema First, navigate to the [data page](https://app.datawisp.io/my-datasources) by clicking on the "Data" button on the left.
Then, for the data source that changed, click the three dots on the very right:
And last, click on "Reload schema" in the menu:
Leave the Datawisp tab open until the process is finished. Usually, this takes less than thirty seconds, but in rare cases this can take up to 10 minutes, depending on the size of your database schema. Once it is done, the "Modified" date of the data source should be the current time, you can also validate that the changes to the schema are visible in Datawisp. ### Updating the data dictionary If the schema has changed, it is often sensible to update the data dictionary. You can update individual tables, or all tables at once by clicking "Generate data dictionary" in the same menu:
If you already have an existing data dictionary, you have two choices:
1. You can **overwrite** your current data dictionary - in that case, all manual changes you've made to the data dictionary will be lost. 2. You can **keep your changes** - in that case, only newly added tables and/or columns will be described by the AI. Already existing entries in the data dictionary will not be touched. If you know exactly what tables changed, it's often faster to just update the data dictionary for those tables. In that case, open up the data dictionary for that specific table:
There, you can press the "Generate with AI" button. # Data Dictionaries Data Dictionaries are a useful way to communicate how the AI and other analysts should work with the data. First, head to the [my data sources](https://app.datawisp.io/my-datasources) tab, by clicking on "Data" in the left menu bar.
Then, expand the data source you want to create data dictionaries for:
Press the "pen" icon to edit the data dictionaries. From there, you should see a screen looking like this:
You can always generate the data dictionaries with AI, by clicking the "Generate with AI" button. You can also manually add information that is unique to your business! ### How to write a good data dictionary You should write the data dictionary so that it would help another person who isn't super familiar with your company to answer questions about your data. You can add the following information * **Table Description**\ This is a free-text field, where you can describe what kind of data is in the table, and how to use it. You can also leave information such as how to join the data with other tables. * **Column Descriptions**\ Describe what kind of information is present in the columns * **Column format**\ If needed, you can describe how the column is formatted. e.g. "string, 'YYYY-MM-DD'" Once you're done, press the "save" button! # Analyzing Data Analyzing data in Datawisp is done by combining a series of blocks that replace code to transform data in specific ways (e.g. filters, aggregations, calculations). Users can either create a blank worksheet and work with the blocks themselves, or ask Wispy (our AI assistant) a question about their data in plain English. Each method has its pros and cons, but both are designed to be well within reach of non-technical users. For beginners, it's recommended to start by asking Wispy a question. Once users gain a little bit of experience with the blocks, they can more confidently create their own sheets. # Datawisp Blocks Datawisp is built on blocks that can be dragged and dropped in place. These blocks represent data sources, transformations to the data, or types of output. Datawisp lets users chain these blocks together to create a sheet that mirrors a data analysis pipeline - this can be a series of joins, filters, aggregations, and more. This section contains an overview of all the major Datawisp blocks and how to use them. # Data Source ![](https://lh7-us.googleusercontent.com/SuHCeGw45T4sdnsff6Jun_ls4jQSZH78VAsMvJqKZvJy-EamJAAUShWdkegur0FeA5l7ULB52lfPgv0vSx7Uz67OYP2EwYQCH2QAnCkmmIWelwLF14La7tbiJS6dlXWtriVoWjkp0YnXgjDyHV5jg54) The data source block is the beginning of analysis. Once your data is imported in or connected to Datawisp, it can be accessed in a Data Source block via the default Data I’ve Imported option in the Data set dropdown. ![](https://lh7-us.googleusercontent.com/L8B_TGT99C0cczwqtPJbJW8T1m0V2hNRG8_V3dHFVa5XQcq7Bd4nSgt7XScKyiwN_YazLPUjbADdzAaOQWdJgfJ7YyL5mFDJwgpI_3SsQhNMum6pFrMz6XDkn_PB8ElKeL2CEhltCJrZwB5nYXVkZt0) \ Once a you’ve selected a Data set, clicking on the block will generate a data preview in the lower part of your screen. You’ll have the ability to see a preview of the data in any block going forward in your analysis by simply selecting the block. In addition to the default Data I’ve Imported option of the Get data from dropdown, there are also some less common options. ![](https://lh7-us.googleusercontent.com/v8UPxop-Ao9a5wlW_6EGEjkm7Wa-7LQ7DtTy0pH-7MNIGeBonEoieKW1AAjkBqEi1EwYX7siD1y2WU8291UxCX0j-B4VWsjUugFz5B8LLBSqbusdpQX3zd1XRS2LXbn3Y0ZWjKhEST6B1LEsms1JX4c) Import new data: This takes you to the data import tool as seen here, allowing you to connect a new data source. ![](https://lh7-us.googleusercontent.com/MCfJ6xmonj1It1WRcsyo9xnOWQyia1iiwzgHw71o0AKHmygLYRoL57HzveTlMecBTsSdY4XG1tifD0D3nxmnOk2SGtZpGaeyS2Oo4JQQ4Zfj6MDlDf6LWU60J0t_1PQusfpjnRcAz1Yr2ErBho_86O8) Datawisp Data: Perform analysis on the default sample datasets Datawisp provides Data from another sheet: Reference a named table built previously in another Datawisp sheet Manually input data: Input data for your analysis manually in a spreadsheet interface. ![](https://lh7-us.googleusercontent.com/sPIWz5Z8JtKQBy7b2KlnDOxf01MhGJBConqm86vGfjw_yBaMjjSAbVBifOQVgfORX3qqkmgBL7B9H03GS26iXY_QB35KbOvyFUj1ZDN66wr2whHqqgMezhcq3u5F-gBMxcANX9StcDqM4dAqYJG7vyU) Generate a series of numbers: Create data by selecting a starting point, end point, and step size. ![](https://lh7-us.googleusercontent.com/hwjrCtiF7GN8j-qSH1mkjcJLsys0c7sLEF3qgWIopp5M1qRTK-5wid-dAtb1kbjtPBaeCvtlz6qvUmCIJaGrhadgmPH3BuYMePN1MU-DdWfxK3AUVpcdQDdzFOMxDxVyKra7ys_qcn_m-MR5gQawoaI)
# Summarize & Group ![](https://lh7-us.googleusercontent.com/0Dl4npXW3Xu_Nd5UsJkk-xU0kRNG77uaMAwOctrOH8Y4vmnoeU1W29fCSxHxTBLdZPqkALZf1K6-9jXvsec-IGNM3dGfYlKv5f40f5MXqyJJySXOuw9BmTdx_T1T89cLObB6kXTVF6_kthviLxrRv4Y) \ Summarize & Group is perhaps Datawisp’s most powerful block, giving the ability to count rows, make groups, calculate summary statistics, and much more. By default, no groups or calculations are selected, so Summarize & Group simply counts the number of rows coming into the block. ![](https://lh7-us.googleusercontent.com/NuSjES_QgGP7k6dI4mLL6EYjTmRZoBMBlGvA-YoNtUltSEi5o6Yf3JfmC_D6QHGtAZhnwOwqfz_PMKyyQrWsyTWLz_XS9R7CP9Nbz4_1IMLr-j21_uFz5v4EMf0hpzCyPRdHWbPta3M2eEAv53aDkm0) Group Data By allows the selection of subgroups. Similar to a SQL Group By statement, any number of columns can be selected and, by default, the counts in those subgroup combinations will be generated. With a single group selected (ex. Primary Fur Color) the count column now shows the number of rows (squirrels) of each Primary Fur Color. Note that the sum of each of these groups would be equal to the original 3,032 above. ![](https://lh7-us.googleusercontent.com/WxubFrb27GYOpc1BJBujZOdsZYhprs8xPG3PEvgaN7Y1sLG4wDpEYUFiNbtfUeJX4YW6-BAow8Ug4ZYbSo8LGV5sXIsGVwmmy-5dcUoPWFIuldqPV4UOAWibnljUFy0RBFs3TCtHn739g-w9B2b5EbM)
When multiple groups are selected, this further breaks down the groups as seen below. ![](https://lh7-us.googleusercontent.com/96-aBhSLL9WKYPkyiIBwQpB4zO5lz1NUM9RoMGt_YWLOxftGAaU-Apk_BJfYkfLTtFXUn-LU1lPC1uptNOZWZL_KozBnufwq9-3jHxUncF5Tad1YxHcx77jVALF6aV5utOwkptsL0ULvT9rc6eAz_dI) \ To add a Value to calculate in addition to just count, additional columns can be chosen here as well. Min, max, sum, and avg of numeric features can be calculated for each group selected. Used together, these summary statistics can summarize large datasets by consolidating them into a few meaningful rows. # Combine Coming soon. # Order and Select Rows Coming soon. # Table Coming soon. # Filter Coming soon. # Add Column Coming soon. # Cleanup Coming soon. # JSON Coming soon. # Chart Coming soon. # Reshape # Wispy Wispy is Datawisp's AI-powered assitant. Users can ask Wispy questions about their data in plain English and get tables, charts, and more. Wispy will create a worksheet for the user and answer the question by connecting the right Datawisp blocks together. Here's a short video showing some of Wispy's capabilities: {% embed url="" %} Wispy launch video {% endembed %} # The SQL View When you ask Wispy a question, you will see alongside the blocks and the results, an **SQL view.**

Datawisp's SQL view

This shows two SQL queries, the "AI SQL" and the "Database SQL". Note: Especially in "thorough" mode, the LLM can sometimes generate more than one query, you can switch between those by clicking on the dots in the chat view on the right. ### Overview In general, the flow is as follows: 1. **LLM writes text, conveys intention** 1. Writes postgresql ("AI SQL") 2. Creates blocks directly 2. **Datawisp** converts to blocks 1. The SQL and other operations are converted to a Datawisp sheet. 3. **Datawisp** executes the query 1. Datawisp chooses to compile (some of) the blocks to the SQL dialect your database understands ("Database SQL") 2. Datawisp executes the query directly using the Datawisp Query Engine ### AI SQL The AI SQL is directly what the LLM generated to analyze your data. To make sure you always get the best experience, we require the LLM to always write postgresql. AI SQL is never run directly on your database. The primary use is for the LLM to convey it's intention and plan for the data analysis. #### Why PostgreSQL Most SQL dialects have some quirks that current-gen LLMs sometimes get wrong. They generate valid, but buggy SQL. Those can lead to unexpected and bad results. To give the best user experience, Datawisp detects and handles many such cases. The LLM is always required to write in one dialect - posgresql. We chose this dialect as it is a rather expressive dialect, has sensible date functions, straight-forward JSON support, and is widely used - so the LLM will have encountered it frequently in training. #### Example A specific example of a SQL dialect quirk: When asked to generate MySQL, sometimes, but not often, the LLM would generate a query like this: ```sql SELECT AVG(arrival_date - ship_date) AS average_shipping_time FROM shipments; ``` This is completely valid MySQL, and it would run without issues. However, the output of (arrival\_date - ship\_date) is rather unexpected: 
MySQL [(none)]> select (CURRENT_DATE);
+----------------+
| (CURRENT_DATE) |
+----------------+
| 2025-03-04     |
+----------------+

MySQL [(none)]> select (CURRENT_DATE - '2025-03-01');
+-------------------------------+
| (CURRENT_DATE - '2025-03-01') |
+-------------------------------+
|                      20248279 |
+-------------------------------+

MySQL [(none)]> select (CURRENT_DATE - '2025-01-01');
+-------------------------------+
| (CURRENT_DATE - '2025-01-01') |
+-------------------------------+
|                      20248279 |
+-------------------------------+

MySQL [(none)]> select (CURRENT_DATE - 0);
+--------------------+
| (CURRENT_DATE - 0) |
+--------------------+
|           20250304 |
+--------------------+
Generated MySQL is fine almost all of the time, but sometimes LLMs make mistakes like this. This happens especially frequently in scenarios that require more complicated SQL. By focusing on one dialect, we can reliably detect and mitigate edge cases like this, and create a consistently good experience, no matter which database is used. ### Database SQL As soon as the LLM finished generating a query, this query is compiled to Datawisp blocks:

The above query as Datawisp blocks

Those blocks are then, where possible, converted to optimized SQL queries that your database engine can run directly. Since this SQL is mainly generated to be interpreted by your database engine, it may sometimes look a bit convoluted. In general, the extraneous parts (e.g. unnecessary `WHERE 1=1`) do not affect the result or the query performance negatively. # Deployment Pick the right deployment for your organization There are several ways to deploy Datawisp based on your security and compliance needs. If you have any questions, reach out to . For more information about how and where your data is processed, please refer to [data processing](/data-processing). ### Shared Deployment - app.datawisp.io *Included in pro plan* The pro plan includes access to the shared deployment of Datawisp. All of Datawisp’s extensive protections for your data apply. For example, data is encrypted at rest, connections are isolated, and strict access controls are enforced. Advantages: * Zero Setup * Instant Support ### Managed Deployment *Included in pro plan for subscribers with 10+ seats* In the cloud of your choice, Datawisp will deploy a separate instance of Datawisp, only accessible to your users. Datawisp will manage the instance for you. Data is encrypted, and tenants are strictly isolated from each other. The deployment will be in the region of your choice, which also can improve loading times. Sign-in and account creation will be limited to domains and login methods of your choice. Advantages: * Zero Setup * Instant Support * Deployment in the region and cloud of your choice. * Data does not leave your cloud provider’s infrastructure.

Managed Deployment - Overview

### Datawisp Enterprise Server (Self-Hosted) *Included in enterprise subscription* Datawisp Enterprise Server is a self-hosted version of the Datawisp platform. Your business can benefit from increased control and avoid issues associated with public clouds, while you can benefit from all features and workflows of Datawisp. Datawisp Enterprise Server is suitable for enterprises that are subject to regulatory compliance. It runs on your infrastructure and is governed by access and security controls that you define, such as firewalls, network policies, IAM, monitoring, and VPNs. If your main compliance requirement is for your company's data to reside in a specific region, you may want to consider the Managed Deployment. With this option, you won't need to schedule downtime for maintenance or upgrades, and your users will have access to the latest features and the full support of Datawisp.

Enterprise Deployment - Overview

#### Deployment Datawisp will provide a pre-configured docker deployment via docker-compose. To provide support for your users, Datawisp will collect sanitized logs. Sign-in and account creation will be limited to domains and login methods of your choice, to prevent unauthorized access. #### Option: Very Low Egress If you have especially stringent requirements (e.g. due to regulatory reasons) , Datawisp can be configured in a low-egress mode. This would stop even sanitized logs being sent to Datawisp. In this case, the support workflow for your users changes. If you are interested in this option, please ask us for details! #### Option: Instant Support Some issues can be resolved instantly by our team. This requires some limited access for some named members of our team to access your instance. In some cases, this reduces friction when resolving issues. # Security How we keep your data secure Keeping your data secure is important. Because of that, Datawisp is built using a secure architecture, and takes advantage of modern, memory-safe languages, and employs additional techniques to prevent any data exfiltration. We are confident that Datawisp lives up to the highest security requirements of your organization. If you have any questions about how Datawisp keeps your data secure, please reach out to . ### Certifications Datawisp holds **SOC 2 Type II** and **SOC 2 Type I** certifications, with zero defects found. For a copy of the detailed reports, please reach out to . The shared instance is hosted in an **ISO 27001** certified datacenter, the managed / enterprise deployments will be hosted in a cloud / data center of your choosing, with all necessary certifications. Datawisp can be operated in a way that complies with HIPAA and other similarly strict certifications - including the AI features. If your business requires a BAA or equivalent, please reach out to our team for options. ### Deployment Options How and where we process data depends on the type of deployment you select. For more details about the possible deployments, refer to our documentation on [deployments](/deployment). * For **enterprise deployments**, your data will never\* leave your infrastructure and control. It remains fully under your control. * For **managed deployments**, your data will never\* your cloud. * For the **shared deployment**, your data is stored in a secure facility with the strictest access controls. \* Depending on your choice of LLM provider. For more details, refer to [data processing](/data-processing). ### Secure Architecture Datawisp is built from ground up with security in mind. * Datawisp does not copy the contents of your databases. Instead, data remains on your infrastructure, and queries are executed on your infrastructure. * Datawisp uses safety-oriented programming languages like Rust and TypeScript, to eliminate entire classes of bugs. * Your data is only stored in memory, and only while you're actively using Datawisp. If you close the relevant tab, your data is automatically evicted from memory. * All data is always encrypted at-rest and in-transit. * The architecture of Datawisp deployments is intentionally kept simple, to eliminate risky complexity. For more detailed information about how we store and process your data, refer to [data processing](/data-processing), and/or ask for a copy of the SOC 2 reports. ### Granular Access Control When you store your data in Datawisp, you can decide what you do with it: * Data is not shared unless you explicitly share it. * Other users will only have access to your data if you explicitly share it with them. * Reviewing who you shared your data with is easily accessible via the sharing menu. ### AI None of the data processed by Datawisp will ever be used to train AI models / large language models. Your data will never be disclosed to other customers of Datawisp and/or the LLM provider. # Privacy At Datawisp, we understand that privacy is one of the most important factors when it comes to selecting a data platform. Datawisp does not use your data for any other purpose than: * Facilitating your use of the platform * Customer service (and only if you explicitly opt into sharing it with us) Datawisp does collect information about how you use the platform to help us continuously improve our offerings. This includes things like: * How often you use Datawisp * Which types of blocks you use * Which blocks are often used in combination with each other For the most correct and up-to-date information on what data we collect and analyze, and what we do with it, please always review our privacy policy and terms of service. Privacy policy: Terms of use: # Data processing How and where is my data processed? For security and compliance reasons, it is often important to understand which infrastructure processes which exact data points when you are using Datawisp. Some of the information below depends on which plan and type of deployment you chose for Datawisp. For more information, check out our [deployment overview](/deployment). ## Primary data processing infrastructure In general, when using Datawisp, your data is processed in four places:
1. **Your existing data warehouse / database**\ This is the primary place where most of the processing takes place. Datawisp processes, whenever possible, everything on your infrastructure, which is completely under your control.\ \ Your existing infrastructure already complies with all of your security / certification requirements.
2. **Your browser**\ In order to display charts, tables, and other visualizations, your browser will process and cache some of the data locally. Most of the data is retained only until you close the browser tab, but some data (e.g. names of tables and columns) may be persisted until you log out.\ \ Your browser’s exact data retention can be controlled and managed by your organization’s IT department, and complies with any security / certification requirements you may have.
3. **Datawisp servers**\ Some processing of the data can happen on the Datawisp server, for example, if you’re trying to combine data from a database with an excel sheet that you’ve uploaded to Datawisp. Except for files which you’ve explicitly uploaded to Datawisp, our servers generally do not retain a copy of your data, and all cached data is destroyed when you close the browser tab.\ \ However, some data points may still be stored by the Datawisp server. For example, the answers Wispy gives are stored in your account, and those may reference specific data points. The data for charts and other visualizations in dashboards are cached to prevent long loading times.\ \ *For customers using the app.datawisp.io*: All of Datawisp’s extensive protections for your data apply. For example, data is encrypted at rest, connections are isolated, and strict access controls are enforced.\ \ *For managed / enterprise deployments*: All processing and storage of your data is controlled and managed by your organization’s infrastructure / IT department, and complies with any security / certification requirements you may have.\ \ For more information, consult our [deployment overview](https://docs.datawisp.io/deployment).
4. **Large Language Models / AI**\ Datawisp minimizes the amount of data sent to LLMs. Datawisp only enables LLM vendors that will *never* use your data or your interactions with Wispy to train future models.\ \ Your data is retained for the minimal period required by your selected vendor, and then deleted.\ \ For a more detailed breakdown, refer to the [Large Language Models](#large-language-models) section below. ### Non data-processing vendors / infrastructure In addition, some usage data is sent to other vendors, as necessary to provide specific features. Those vendors do not process or get access to your data. All information they would receive about your data would be incidental. Nonetheless, in enterprise / managed deployments, those features can be turned off. 1. **AI Voice Input** (Gladia)\ For voice input, your browser may send voice recordings directly to gladia.io. This only happens if you press the “microphone” icon in the text input fields.\ \ *For managed / enterprise deployments*, this feature can be turned off.
2. **Log collection** (Honeycomb, Sentry, Datadog, Datawisp)\ By default, Datawisp collects some usage information via Honeycomb, Sentry, and Datadog. The Datawisp server also stores some usage logs. Logs generally do not contain specific data points. However, they may sometimes reference table or column names.\ \ *For managed / enterprise deployments*, automatic log collection can be turned off. In that case, individual server- and client log collections can be sent to the Datawisp team to resolve potential issues.
3. **Login:** Your respective SSO provider (Google, Microsoft, LinkedIn, GitHub, Stytch)\ If you are logging in using a third party, Datawisp verifies which user is trying to authenticate with them. From this information, they could, for example, derive when and how often you use Datawisp. If you are logging in with email / password, the relevant provider is Stytch.\ \ *For managed / enterprise deployments*, we recommend only enabling your existing / trusted SSO provider. ## Large Language Models To enable Wispy to analyze your data, create charts, give answers and create dashboards, Datawisp leverages large language models. ### Supported Vendors / Models The current generation of state-of-the-art large language models are often proprietary and/or difficult to deploy. Therefore, the best models for Datawisp are generally not deployable directly on your infrastructure. If you do have access to a private instance of any of the supported models, Datawisp can, of course, connect to those. By default, Datawisp currently supports models provided by: * **Microsoft Azure** (GPT) * **OpenAI** (GPT) **Note:** Your data or chats are never used to train AI, ever. In general, data is retained at most for 30 days. Both Azure and OpenAI have some options for zero-retention. For exact and up-to-date information, refer to the respective vendors. ([Azure](https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy?tabs=azure-portal), [OpenAI](https://platform.openai.com/docs/guides/your-data)) Specifically, for organizations with strict security and/or compliance needs, we recommend using models available via Microsoft Azure. Azure has all necessary certifications (incl. HIPAA, SOC2, …), and is trusted by the largest organizations around the world. ### Other Models If neither Microsoft Azure nor OpenAI deployments are a possibility for your organization, other model providers can be explored, for example: * Google Cloud (Gemini) * AWS (Claude, DeepSeek, llama, …) * Anthropic (Claude) * Self-hosted (DeepSeek, llama, …) In general, for enterprise customers, Datawisp can be set up to work with the mentioned model families. However, sometimes this may have implications for performance or accuracy. This is not always because those models are “worse”. In many cases, it is because Datawisp has been more extensively tested with and optimized for the quirks and behaviors of the GPT-family of models. If another vendor is chosen, the relevant processing and retention terms by the vendor would apply. (e.g. [Google Cloud](https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance), [AWS](https://aws.amazon.com/bedrock/security-compliance/), …) Some of those models are also available as “open weight” models, and can be self-hosted by your infrastructure team. In those cases, please be prepared to host the best performing / biggest model of the respective model family. ### What data is transmitted to the LLM? To make sure Wispy gives the best answer possible, Datawisp carefully balances the amount of data that is sent to the LLM. For best performance, LLMs, like human data analysts, do need to be able to see some data to understand its layout, or summarize the results of queries. However, sharing too much data is undesirable for both accuracy and security. Therefore, Datawisp often limits the amount of data it sends to the LLM In general, Datawisp may transmit some of the following data to the LLM you selected: * the prompt / question entered by the user * your organization's global prompt / memories * informations about the selected tables (names, column names, data types, relations, data dictionaries, ...) * a small, random sample of rows from those tables * the first few rows of each result set for queries the AI writes However, as Wispy improves, the exact makeup may change. Any subscription of Datawisp has access to our AI inspector. Using that, your team can see exactly what kind of your data specifically was sent to the LLM for any prompt by the user, and what model was used.

Datawisp's AI inspector

In general, our recommendation is to choose a trustworthy LLM provider, instead of trying to minimize which exact data points are transmitted. The correct amount of sensitive data to send a LLM provider you can not trust is, unfortunately, zero. # Contact Info For more information or to set up a meeting with the Datawisp team, please reach out at: . You can also join our public support [Discord channel](https://discord.gg/8Jkwh2u2Wp). # Additional Resources Our official website: Request a demo: Try it for yourself (free):[ https://app.datawisp.io/](https://app.datawisp.io/) Pricing info: \ Our blog: Twitter: LinkedIn: YouTube:
# Tutorials {% embed url="" %} More coming soon.