coderkun/questlab
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Wiki Activity

update Piwik to version 2.16 (fixes #91)

Browse source
This commit is contained in:
oliver 2016-04-10 18:55:57 +02:00
parent 296343bf3b
commit d885a4baa9
5833 changed files with 418669 additions and 226797 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.hgignore
View file

@ -6,5 +6,5 @@ syntax: regexp
^seminarymedia/*
^seminaryuploads/*
^www/analytics/config/config.ini.php*
^www/analytics/temp/*
^www/analytics/tmp/*
^app/lib/phpqrcode/cache/*

6
views/html/html.tpl
View file

@ -128,6 +128,7 @@
<?=$seminarybar?>
<?php endif ?>
</aside>
<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
_paq.push(['trackPageView']);
@ -136,11 +137,12 @@
var u=(("https:" == document.location.protocol) ? "https" : "http") + "://" + document.location.hostname + "/analytics/";
_paq.push(['setTrackerUrl', u+'piwik.php']);
_paq.push(['setSiteId', 1]);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.type='text/javascript';
g.defer=true; g.async=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script>
<noscript><p><img src="/analytics/piwik.php?idsite=1" style="border:0;" alt="" /></p></noscript>
<!-- End Piwik Code -->
</body>
</html>

411
www/analytics/CHANGELOG.md Normal file
View file

@ -0,0 +1,411 @@
# Piwik Platform Changelog
This is a changelog for Piwik platform developers. All changes for our HTTP API's, Plugins, Themes, etc will be listed here.
## Piwik 2.16.0
### New features
* New segment `actionType` lets you segment all actions of a given type, eg. `actionType==events` or `actionType==downloads`. Action types values are: `pageviews`, `contents`, `sitesearches`, `events`, `outlinks`, `downloads`
* New segment `actionUrl` lets you segment any action that matches a given URL, whether they are Pageviews, Site searches, Contents, Downloads or Events.
* New segment `deviceBrand` lets you restrict your users to those using a particular device brand such as Apple, Samsung, LG, Google, Nokia, Sony, Lenovo, Alcatel, etc. View the [complete list of device brands.](http://developer.piwik.org/api-reference/segmentation)
* New segment operators `=^` "Starts with" and `=$` "Ends with" complement the existing segment operators: Contains, Does not contain, Equals, Not equals, Greater than or equal to, Less than or equal to.
* The JavaScript Tracker method `PiwikTracker.setDomains()` can now handle paths. This means when setting eg `_paq.push(['setDomains, '*.piwik.org/website1'])` all link that goes to the same domain `piwik.org` but to any other path than `website1/*` will be treated as outlink.
* In Administration > Websites, for each website, there is a checkbox "Only track visits and actions when the action URL starts with one of the above URLs". In Piwik 2.14.0, any action URL starting with one of the Alias URLs or starting with a subdomain of the Alias URL would be tracked. As of Piwik 2.15.0, when this checkbox is enabled, it may track less data: action URLs on an Alias URL subdomain will not be tracked anymore (you must specify each sub-domain as Alias URL).
* It is now possible to pass an option `php-cli-options` to the `core:archive` command. The given cli options will be forwarded to the actual PHP command. This allows to for example specifiy a different memory limit for the archiving process like this: `./console core:archive --php-cli-options="-d memory_limit=8G"`
* New less variable `@theme-color-menu-contrast-textSelected` that lets you specify the color of a selected menu item.
* in Administration > Diagnostics, there is a new page `Config file` which lets Super User view all config values from `global.ini.php` in the UI, and whether they were overriden in your `config/config.ini.php`
### New commands
* New command `config:set` lets you set INI config options from the command line. This command can be used for convenience or for automation.
### Internal changes
* `UsersManager.*` API calls: when an API request specifies a `token_auth` of a user with `admin` permission, the returned dataset will not include all usernames as previously, API will now only return usernames for users with `view` or `admin` permission to website(s) viewable by this `token_auth`.
* When generating a new plugin skeleton via `generate:plugin` command, plugin name must now contain only letters and numbers.
* JavaScript Tracker tests no longer require `SQLite`. The existing MySQL configuration for tests is used now. In order to run the tests make sure Piwik is installed and `[database_tests]` is configured in `config/config.ini.php`.
* The definitions for search engine and social network detection have been moved from bundled data files to a separate package (see [https://github.com/piwik/searchengine-and-social-list](https://github.com/piwik/searchengine-and-social-list)).
* In [UI screenshot tests](https://developer.piwik.org/guides/tests-ui), a test environment `configOverride` setting should be no longer overwritten. Instead new values should be added to the existing `configOverride` array in PHP or JavaScript. For example instead of `testEnvironment.configOverride = {group: {name: 1}}` use `testEnvironment.overrideConfig('group', 'name', '1')`.
### New APIs
* Add your own SMS/Text provider by creating a new class in the `SMSProvider` directory of your plugin. The class has to extend `Piwik\Plugins\MobileMessaging\SMSProvider` and implement the required methods.
* Segments can now be composed by a union of multiple segments. To do this set an array of segments that shall be used for that segment `$segment->setUnionOfSegments(array('outlinkUrl', 'downloadUrl'))` instead of defining a SQL column.
### Deprecations
* The method `DB::tableExists` was un-used and has been removed.
## Piwik 2.15.0
### New commands
* New command `diagnostics:analyze-archive-table` that analyzes archive tables
* New command `database:optimize-archive-tables` to optimize archive tables and possibly save disk space (even if on InnoDB)
* New Command `core:invalidate-report-data` to invalidate archive data (w/ period cascading) ([FAQ](https://piwik.org/faq/how-to/faq_155/))
### New APIs and features
* Piwik 2.15.0 is now mostly compatible with PHP7.
* The JavaScript Tracker `piwik.js` got a new method `logAllContentBlocksOnPage` to log all found content blocks within a page to the console. This is useful to debug / test content tracking. It can be triggered via `_paq.push(['logAllContentBlocksOnPage'])`
* The Class `Piwik\Plugins\Login\Controller` is now considered a public API.
* The new method `Piwik\Menu\MenuAbstract::registerMenuIcon()` can be used to define an icon for a menu category to replace the default arrow icon.
* New event `CronArchive.getIdSitesNotUsingTracker` that allows you to set a list of idSites that do not use the Tracker API to make sure we archive these sites if needed.
* New events `CronArchive.init.start` which is triggered when the CLI archiver starts and `CronArchive.end` when the archiver ended.
* Piwik tracker can now be configured with strict Content Security Policy ([CSP FAQ](https://piwik.org/faq/general/faq_20904/)).
* Super Users can choose whether to use the latest stable release or latest Long Term Support release.
### Breaking Changes
* The method `Dimension::getId()` has been set as `final`. It is not allowed to overwrite this method.
* We fixed a bug where the API method `Sites.getPatternMatchSites` only returned a very limited number of websites by default. We now return all websites by default unless a limit is specified specifically.
* Handling of localized date, time and range formats has been changed. Patterns no longer contain placeholders like %shortDay%, but work with CLDR pattern instead. You can use one of the predefined format constants in Date class for using getLocalized().
* As we are now using CLDR formats for all languages, some time formats were even changed in english. Attributes like prettyDate in API responses might so have been changed slightly.
* The config `enable_measure_piwik_usage_in_idsite` which is used to track the Piwik usage with Piwik was removed and replaced by a new plugin `AnonymousPiwikUsageMeasurement`
### Deprecations
* The following HTTP API methods have been deprecated and will be removed in Piwik 3.0:
* `SitesManager.getSitesIdWithVisits`
* `API.getLastDate`
* The following events have been deprecated and will be removed in Piwik 3.0. Use [dimensions](http://developer.piwik.org/guides/dimensions) instead.
* `Tracker.existingVisitInformation`
* `Tracker.getVisitFieldsToPersist`
* `Tracker.newConversionInformation`
* `Tracker.newVisitorInformation`
* `Tracker.recordAction`
* `Tracker.recordEcommerceGoal`
* `Tracker.recordStandardGoals`
* The Platform API method `\Piwik\Plugin::getListHooksRegistered()` has been deprecated and will be removed in Piwik 3.0. Use `\Piwik\Plugin::registerEvents()` instead.
### Internal changes
* When logging in, the username is now case insensitive
* URLs with emojis and any other unicode character will be tracked, with special characters replaced with `<60>`
* A permanent warning notification is now displayed when PHP is 5.4.* or older, since it has reached End Of Life
* In `piwik.js` we replaced [JSON2](https://github.com/douglascrockford/JSON-js) with [JSON3](https://bestiejs.github.io/json3/) to implement CSP (Content Security Policy) as JSON3 does not use `eval()`. JSON3 will be used if a browser does not provide a native JSON API. We are using `JSON3` in a way that it will not conflict if your website is using `JSON3` as well.
* The option `branch` of the console command `development:sync-system-test-processed` was removed as it is no longer needed.
* All numbers in reports will now appear formatted (eg. `1,000,000` instead of `1000000`)
* Database connections now use `UTF-8` charset explicitely to force UTF-8 data handling
## Piwik 2.14.0
### Breaking Changes
* The `UserSettings` API has been removed. The API was deprecated in earlier versions. Use `DevicesDetection`, `Resolution` and `DevicePlugins` API instead.
* Many translations have been moved to the new Intl plugin. Most of them will still work, but please update their usage. See https://github.com/piwik/piwik/pull/8101 for a full list
### New features
* The JavaScript Tracker does now track outlinks and downloads if a user opens the context menu if the `enabled` parameter of the `enableLinkTracking()` method is set to `true`. To use this new feature use `tracker.enableLinkTracking(true)` or `_paq.push(['enableLinkTracking', true]);`. This is not industry standard and is vulnerable to false positives since not every user will select "Open in a new tab" when the context menu is shown. Most users will do though and it will lead to more accurate results in most cases.
* The JavaScript Tracker now contains the 'heart beat' feature which can be used to obtain more accurate visit lengths by periodically sending 'ping' requests to Piwik. To use this feature use `tracker.enableHeartBeatTimer();` or `_paq.push(['enableHeartBeatTimer']);`. By default, a ping request will be sent every 15 seconds. You can specify a custom ping delay (in seconds) by passing an argument, eg, `tracker.enableHeartBeatTimer(10);` or `_paq.push(['enableHeartBeatTimer', 10]);`.
* New custom segment `languageCode` that lets you segment visitors that are using a particular language. Example values: `de`, `fr`, `en-gb`, `zh-cn`, etc.
* Segment `userId` now supports any segment operator (previously only operator Contains `=@` was supported for this segment).
### Commands updates
* The command `core:archive` now has two new parameter: `--force-idsegments` and `--skip-idsegments` that let you force (or skip) processing archives for one or several custom segments.
* The command `scheduled-tasks:run` now has an argument `task` that lets you force run a particular scheduled task.
### Library updates
* Updated pChart library from 2.1.3 to 2.1.4. The files were moved from the directory `libs/pChart2.1.3` to `libs/pChart`
### Internal change
* To execute UI tests "ImageMagick" is now required.
* The Q JavaScript promise library is now distributed with tests and can be used in the piwik.js tests.
## Piwik 2.13.0
### Breaking Changes
* The API method `Live.getLastVisitsDetails` does no longer support the API parameter `filter_sort_column` to prevent possible memory issues when `filter_offset` is large.
* The Event `Site.setSite` was removed as it causes performance problems.
* `piwik.php` does now return a HTTP 400 (Bad request) if requested without any tracking parameters (GET/POST). If you still want to use `piwik.php` for checks please use `piwik.php?rec=0`.
### Deprecations
* The method `Piwik\Archive::getBlob()` has been deprecated and will be removed from June 1st 2015. Use one of the methods `getDataTable*()` methods instead.
* The API parameter `countVisitorsToFetch` of the API method `Live.getLastVisitsDetails` has been deprecated as `filter_offset` and `filter_limit` work correctly now.
### New commands
* There is now a `diagnostic:run` command to run the system check from the command line.
* There is now an option `--xhprof` that can be used with any command to profile that command via XHProf.
### APIs Improvements
* Visitor details now additionally contain: `deviceTypeIcon`, `deviceBrand` and `deviceModel`
* In 2.6.0 we added the possibility to use `filter_limit` and `filter_offset` if an API returns an indexed array. This was not working in all cases and is fixed now.
* The API parameter `filter_pattern` and `filter_offset[]` can now be used if an API returns an indexed array.
### Internal changes
* The referrer spam filter has moved from the `referrer_urls_spam` INI option (in `global.ini.php`) to a separate package (see [https://github.com/piwik/referrer-spam-blacklist](https://github.com/piwik/referrer-spam-blacklist)).
## Piwik 2.12.0
### Breaking Changes
* The deprecated method `Period::factory()` has been removed. Use `Period\Factory` instead.
* The deprecated method `Config::getConfigSuperUserForBackwardCompatibility()` has been removed.
* The deprecated methods `MenuAdmin::addEntry()` and `MenuAdmin::removeEntry()` have been removed. Use `Piwik\Plugin\Menu` instead.
* The deprecated methods `MenuTop::addEntry()` and `MenuTop::removeEntry()` have been removed. Use `Piwik\Plugin\Menu` instead.
* The deprecated method `SettingsPiwik::rewriteTmpPathWithInstanceId()` has been removed.
* The following deprecated methods from the `Piwik\IP` class have been removed, use `Piwik\Network\IP` instead:
* `sanitizeIp()`
* `sanitizeIpRange()`
* `P2N()`
* `N2P()`
* `prettyPrint()`
* `isIPv4()`
* `long2ip()`
* `isIPv6()`
* `isMappedIPv4()`
* `getIPv4FromMappedIPv6()`
* `getIpsForRange()`
* `isIpInRange()`
* `getHostByAddr()`
### Deprecations
* `API` classes should no longer have a protected constructor. Classes with a protected constructor will generate a notice in the logs and should expose a public constructor instead.
* Update classes should not declare static `getSql()` and `update()` methods anymore. It is still supported to use those, but developers should instead override the `Updates::getMigrationQueries()` and `Updates::doUpdate()` instance methods.
### New features
* `API` classes can now use dependency injection in their constructor to inject other instances.
### New commands
* There is now a command `core:purge-old-archive-data` that can be used to manually purge temporary, error-ed and invalidated archives from one or more archive tables.
* There is now a command `usercountry:attribute` that can be used to re-attribute geolocated location data to existing visits and conversions. If you have visits that were tracked before setting up GeoIP, you can use this command to add location data to them.
## Piwik 2.11.0
### Breaking Changes
* The event `User.getLanguage` has been removed.
* The following deprecated event has been removed: `TaskScheduler.getScheduledTasks`
* Special handling for operating system `Windows` has been removed. Like other operating systems all versions will now only be reported as `Windows` with versions like `XP`, `7`, `8`, etc.
* Reporting for operating systems has been adjusted to report information according to browser information. Visitor details now contain: `operatingSystemName`, `operatingSystemIcon`, `operatingSystemCode` and `operatingSystemVersion`
### Deprecations
* The following methods have been deprecated in favor of the new `Piwik\Intl` component:
* `Piwik\Common::getContinentsList()`: use `RegionDataProvider::getContinentList()` instead
* `Piwik\Common::getCountriesList()`: use `RegionDataProvider::getCountryList()` instead
* `Piwik\Common::getLanguagesList()`: use `LanguageDataProvider::getLanguageList()` instead
* `Piwik\Common::getLanguageToCountryList()`: use `LanguageDataProvider::getLanguageToCountryList()` instead
* `Piwik\Metrics\Formatter::getCurrencyList()`: use `CurrencyDataProvider::getCurrencyList()` instead
* The `Piwik\Translate` class has been deprecated in favor of `Piwik\Translation\Translator`.
* The `core:plugin` console has been deprecated in favor of the new `plugin:list`, `plugin:activate` and `plugin:deactivate` commands
* The following classes have been deprecated:
* `Piwik\TaskScheduler`: use `Piwik\Scheduler\Scheduler` instead
* `Piwik\ScheduledTask`: use `Piwik\Scheduler\Task` instead
* The API method `UserSettings.getLanguage` is deprecated and will be removed from May 1st 2015. Use `UserLanguage.getLanguage` instead
* The API method `UserSettings.getLanguageCode` is deprecated and will be removed from May 1st 2015. Use `UserLanguage.getLanguageCode` instead
* The `Piwik\Registry` class has been deprecated in favor of using the container:
* `Registry::get('auth')` should be replaced with `StaticContainer::get('Piwik\Auth')`
* `Registry::set('auth', $auth)` should be replaced with `StaticContainer::getContainer()->set('Piwik\Auth', $auth)`
### New features
* You can now generate UI / screenshot tests using the command `generate:test`
* During UI tests we do now add a CSS class to the HTML element called `uiTest`. This allows you do hide content when screenshots are captured.
### New commands
* A new command (core:fix-duplicate-log-actions) has been added which can be used to remove duplicate actions and correct references to them in other tables. Duplicates were caused by this bug: [#6436](https://github.com/piwik/piwik/issues/6436)
### Library updates
* Updated AngularJS from 1.2.26 to 1.2.28
* Updated piwik/device-detector from 2.8 to 3.0
### Internal change
* UI specs were moved from `tests/PHPUnit/UI` to `tests/UI`. We also moved the UI specs directly into the Piwik repository meaning the [piwik-ui-tests](https://github.com/piwik/piwik-ui-tests) repository contains only the expected screenshots from now on.
* There is a new command `development:sync-system-test-processed` for core developers that allows you to copy processed test results from travis to your local dev environment.
## Piwik 2.10.0
### Breaking Changes
* API responses containing visitor information will no longer contain the fields `screenType` and `screenTypeIcon` as those reports have been completely removed
* os, browser and browser plugin icons are now located in the DevicesDetection and DevicePlugins plugin. If you are not using the Reporting or Metadata API to get the icon locations please update your paths.
* The deprecated method `Piwik\SettingsPiwik::rewriteTmpPathWithHostname()` has been removed.
* The following events have been removed:
* `Log.formatFileMessage`
* `Log.formatDatabaseMessage`
* `Log.formatScreenMessage`
* These events have been removed as Piwik now uses the Monolog logging library. [Learn more.](http://developer.piwik.org/guides/logging)
* The event `Log.getAvailableWriters` has been removed: to add custom log backends, you now need to configure Monolog handlers
* The INI options `log_only_when_cli` and `log_only_when_debug_parameter` have been removed
### Library updates
* We added the `symfony/var-dumper` library allowing you to better print any arbitrary PHP variable via `dump($var1, $var2, ...)`.
* Piwik now uses [Monolog](https://github.com/Seldaek/monolog) as a logger.
* The tracker proxy (previously in `misc/proxy-hide-piwik-url/`) has been moved to a separate repository: [https://github.com/piwik/tracker-proxy](https://github.com/piwik/tracker-proxy).
### Deprecations
* Some duplicate reports from UserSettings plugin have been removed. Widget URLs for those reports will still work till May 1st 2015. Please update those to the new reports of DevicesDetection plugin.
* The API method `UserSettings.getBrowserVersion` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowserVersions` instead
* The API method `UserSettings.getBrowser` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowsers` instead
* The API method `UserSettings.getOSFamily` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getOsFamilies` instead
* The API method `UserSettings.getOS` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getOsVersions` instead
* The API method `UserSettings.getMobileVsDesktop` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getType` instead
* The API method `UserSettings.getBrowserType` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowserEngines` instead
* The API method `UserSettings.getResolution` is deprecated and will be removed from May 1st 2015. Use `Resolution.getResolution` instead
* The API method `UserSettings.getConfiguration` is deprecated and will be removed from May 1st 2015. Use `Resolution.getConfiguration` instead
* The API method `UserSettings.getPlugin` is deprecated and will be removed from May 1st 2015. Use `DevicePlugins.getPlugin` instead
* The API method `UserSettings.getWideScreen` has been removed. Use `UserSettings.getScreenType` instead.
* `Piwik\SettingsPiwik::rewriteTmpPathWithInstanceId()` has been deprecated. Instead of hardcoding the `tmp/` path everywhere in the codebase and then calling `rewriteTmpPathWithInstanceId()`, developers should get the `path.tmp` configuration value from the DI container (e.g. `StaticContainer::getContainer()->get('path.tmp')`).
* The method `Piwik\Log::setLogLevel()` has been deprecated
* The method `Piwik\Log::getLogLevel()` has been deprecated
## Piwik 2.9.1
### Breaking Changes
* The HTTP Tracker API does now respond with a HTTP 400 instead of a HTTP 500 in case an invalid `idsite` is used
### New APIs
* New URL parameter `send_image=0` in the [HTTP Tracking API](http://developer.piwik.org/api-reference/tracking-api) to receive a HTTP 204 response code instead of a GIF image. This improves performance and can fix errors if images are not allowed to be obtained directly (eg Chrome Apps).
### New commands
* `core:plugin list` lists all plugins currently activated in Piwik.
## Piwik 2.9.0
### Breaking Changes
* Development related [console commands](http://developer.piwik.org/guides/piwik-on-the-command-line) are only available if the development mode is enabled. To enable the development mode execute `./console development:enable`.
* The command `php console core:update` does no longer have a parameter `--dry-run`. A dry run is now executed by default followed by a question whether one actually wants to execute the updates. To skip this confirmation step one can use the `--yes` option.
### Deprecations
* Most methods of `Piwik\IP` have been deprecated in favor of the new [piwik/network](https://github.com/piwik/component-network) component.
* The file `tests/PHPUnit/phpunit.xml` is no longer needed in order to run tests and we suggest to delete it. The test configuration is now done automatically if possible. In case the tests do no longer work check out the `[tests]` section in `config/global.ini.php`
### Library updates
* Code for manipulating IP addresses has been moved to a separate standalone component: [piwik/network](https://github.com/piwik/component-network). Backward compatibility is kept in Piwik core.
## Piwik 2.8.2
### Library updates
* Updated AngularJS from 1.2.25 to 1.2.26
* Updated jQuery from 1.11.0 to 1.11.1
## Piwik 2.8.0
### Breaking Changes
* The Auth interface has been modified, existing Auth implementations will have to be modified. Changes include:
* The initSession method has been moved. Since this behavior must be executed for every Auth implementation, it has been put into a new class: SessionInitializer.
If your Auth implementation implements its own session logic you will have to extend and override SessionInitializer.
* The following methods have been added: setPassword, setPasswordHash, getTokenAuthSecret and getLogin.
* Clarifying semantics of each method and what they must support and can support.
* **Read the documentation for the [Auth interface](http://developer.piwik.org/api-reference/Piwik/Auth) to learn more.**
* The `Piwik\Unzip\*` classes have been extracted out of the Piwik repository into a separate component named [Decompress](https://github.com/piwik/component-decompress).
* `Piwik\Unzip` has not moved, it is kept for backward compatibility. If you have been using that class, you don't need to change anything.
* The `Piwik\Unzip\*` classes (Tar, PclZip, Gzip, ZipArchive) have moved to the `Piwik\Decompress\*` namespace (inside the new repository).
* `Piwik\Unzip\UncompressInterface` has been moved and renamed to `Piwik\Decompress\DecompressInterface` (inside the new repository).
### Deprecations
* The `Piwik::setUserHasSuperUserAccess` method is deprecated, instead use Access::doAsSuperUser. This method will ensure that super user access is properly rescinded after the callback finishes.
* The class `\IntegrationTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\SystemTestCase` instead.
* The class `\DatabaseTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\IntegrationTestCase` instead.
* The class `\BenchmarkTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\BenchmarkTestCase` instead.
* The class `\ConsoleCommandTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\ConsoleCommandTestCase` instead.
* The class `\FakeAccess` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\Mock\FakeAccess` instead.
* The class `\Piwik\Tests\Fixture` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\Fixture` instead.
* The class `\Piwik\Tests\OverrideLogin` is deprecated and will be removed from February 6ths 2015. Use `\Piwik\Framework\Framework\OverrideLogin` instead.
### New API Features
* The pivotBy and related query parameters can be used to pivot reports by another dimension. Read more about the new query parameters [here](http://developer.piwik.org/api-reference/reporting-api#optional-api-parameters).
### Library updates
* Updated AngularJS from 1.2.13 to 1.2.25
### New commands
* `generate:angular-directive` Let's you easily generate a template for a new angular directive for any plugin.
### Internal change
* Piwik 2.8.0 now requires PHP >= 5.3.3.
* If you use an older PHP version, please upgrade now to the latest PHP so you can enjoy improvements and security fixes in Piwik.
## Piwik 2.7.0
### Reporting APIs
* Several APIs will now expose a new metric `nb_users` which measures the number of unique users when a [User ID](http://piwik.org/docs/user-id/) is set.
* New APIs have been added for [Content Tracking](http://piwik.org/docs/content-tracking/) feature: Contents.getContentNames, Contents.getContentPieces
### Deprecations
* The `Piwik\Menu\MenuAbstract::add()` method is deprecated in favor of `addItem()`. Read more about this here: [#6140](https://github.com/piwik/piwik/issues/6140). We do not plan to remove the deprecated method before Piwik 3.0.
### New APIs
* It is now easier to generate the URL for a menu item see [#6140](https://github.com/piwik/piwik/issues/6140), [urlForDefaultAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlfordefaultaction), [urlForAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlforaction), [urlForModuleAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlformoduleaction)
### New commands
* `core:clear-caches` Lets you easily delete all caches. This command can be useful for instance after updating Piwik files manually.
## Piwik 2.6.0
### Deprecations
* The `'json'` API format is considered deprecated. We ask all new code to use the `'json2'` format. Eventually when Piwik 3.0 is released the `'json'` format will be replaced with `'json2'`. Differences in the json2 format include:
* A bug in JSON formatting was fixed so API methods that return simple associative arrays like `array('name' => 'value', 'name2' => 'value2')` will now appear correctly as `{"name":"value","name2":"value2"}` in JSON API output instead of `[{"name":"value","name2":"value2"}]`. API methods like **SitesManager.getSiteFromId** & **UsersManager.getUser** are affected.
#### Reporting API
* If an API returns an indexed array, it is now possible to use `filter_limit` and `filter_offset`. This was before only possible if an API returned a DataTable.
* The Live API now returns only visitor information of activated plugins. So if for instance the Referrers plugin is deactivated a visitor won't contain any referrers related properties. This is a bugfix as the API was crashing before if some core plugins were deactivated. Affected methods are for instance `getLastVisitDetails` or `getVisitorProfile`. If all core plugins are enabled as by default there will be no change at all except the order of the properties within one visitor.
### New commands
* `core:run-scheduled-tasks` Let's you run all scheduled tasks due to run at this time. Useful for instance when testing tasks.
#### Internal change
* We removed our own autoloader that was used to load Piwik files in favor of the composer autoloader which we already have been using for some libraries. This means the file `core/Loader.php` will no longer exist. In case you are using Piwik from Git make sure to run `php composer.phar self-update && php composer.phar install` to make your Piwik work again. Also make sure to no longer include `core/Loader.php` in case it is used in any custom script.
* We do no longer store the list of plugins that are used during tracking in the config file. They are dynamically detect instead. The detection of a tracker plugin works the same as before. A plugin has to either listen to any `Tracker.*` or `Request.initAuthenticationObject` event or it has to define dimensions in order to be detected as a tracker plugin.
## Piwik 2.5.0
### Breaking Changes
* Javascript Tracking API: if you are using `getCustomVariable` function to access custom variables values that were set on previous page views, you now must also call `storeCustomVariablesInCookie` before the first call to `trackPageView`. Read more about [Javascript Tracking here](http://developer.piwik.org/api-reference/tracking-javascript).
* The [settings](http://developer.piwik.org/guides/piwik-configuration) API will receive the actual entered value and will no longer convert characters like `&` to `&amp;`. If you still want this behavior - for instance to prevent XSS - you can define a filter by setting the `transform` property like this:
`$setting->transform = function ($value) { return Common::sanitizeInputValue($value); }`
* Config setting `disable_merged_assets` moved from `Debug` section to `Development`. The updater will automatically change the section for you.
* `API.getRowEvolution` will throw an exception if a report is requested that does not have a dimension, for instance `VisitsSummary.get`. This is a fix as an invalid format was returned before see [#5951](https://github.com/piwik/piwik/issues/5951)
* `MultiSites.getAll` returns from now on always an array of websites. In the past it returned a single object and it didn't contain all properties in case only one website was found which was a bug see [#5987](https://github.com/piwik/piwik/issues/5987)
### Deprecations
The following events are considered as deprecated and the new structure should be used in the future. We have not scheduled when those events will be removed but probably in Piwik 3.0 which is not scheduled yet and won't be soon. New features will be added only to the new classes.
* `API.getReportMetadata`, `API.getSegmentDimensionMetadata`, `Goals.getReportsWithGoalMetrics`, `ViewDataTable.configure`, `ViewDataTable.getDefaultType`: use [Report](http://developer.piwik.org/api-reference/Piwik/Plugin/Report) class instead to define new reports. There is an updated guide as well [Part1](http://developer.piwik.org/guides/getting-started-part-1)
* `WidgetsList.addWidgets`: use [Widgets](http://developer.piwik.org/api-reference/Piwik/Plugin/Widgets) class instead to define new widgets
* `Menu.Admin.addItems`, `Menu.Reporting.addItems`, `Menu.Top.addItems`: use [Menu](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu) class instead
* `TaskScheduler.getScheduledTasks`: use [Tasks](http://developer.piwik.org/api-reference/Piwik/Plugin/Tasks) class instead to define new tasks
* `Tracker.recordEcommerceGoal`, `Tracker.recordStandardGoals`, `Tracker.newConversionInformation`: use [Conversion Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ConversionDimension) class instead
* `Tracker.existingVisitInformation`, `Tracker.newVisitorInformation`, `Tracker.getVisitFieldsToPersist`: use [Visit Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/VisitDimension) class instead
* `ViewDataTable.addViewDataTable`: This event is no longer needed. Visualizations are automatically discovered if they are placed within a `Visualizations` directory inside the plugin.
### New features
#### Translation search
As a plugin developer you might want to reuse existing translation keys. You can now find all available translations and translation keys by opening the page "Settings => Development:Translation search" in your Piwik installation. Read more about [internationalization](http://developer.piwik.org/guides/internationalization) here.
#### Reporting API
It is now possible to use the `filter_sort_column` parameter when requesting `Live.getLastVisitDetails`. For instance `&filter_sort_column=visitCount`.
#### @since annotation
We are using `@since` annotations in case we are introducing new API's to make it easy to see in which Piwik version a new method was added. This information is now displayed in the [Classes API-Reference](http://developer.piwik.org/api-reference/classes).
### New APIs
* [Report](http://developer.piwik.org/api-reference/Piwik/Plugin/Report) to add a new report
* [Action Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ActionDimension) to add a dimension that tracks action related information
* [Visit Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/VisitDimension) to add a dimension that tracks visit related information
* [Conversion Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ConversionDimension) to add a dimension that tracks conversion related information
* [Dimension](http://developer.piwik.org/api-reference/Piwik/Columns/Dimension) to add a basic non tracking dimension that can be used in `Reports`
* [Widgets](http://developer.piwik.org/api-reference/Piwik/Plugin/Widgets) to add or modfiy widgets
* These Menu classes got new methods that make it easier to add new items to a specific section
* [MenuAdmin](http://developer.piwik.org/api-reference/Piwik/Menu/MenuAdmin) to add or modify admin menu items.
* [MenuReporting](http://developer.piwik.org/api-reference/Piwik/Menu/MenuReporting) to add or modify reporting menu items
* [MenuUser](http://developer.piwik.org/api-reference/Piwik/Menu/MenuUser) to add or modify user menu items
* [Tasks](http://developer.piwik.org/api-reference/Piwik/Plugin/Tasks) to add scheduled tasks
### New commands
* `generate:theme` Let's you easily generate a new theme and customize colors, see the [Theming guide](http://developer.piwik.org/guides/theming)
* `generate:update` Let's you generate an update file
* `generate:report` Let's you generate a report
* `generate:dimension` Let's you enhance the tracking by adding new dimensions
* `generate:menu` Let's you generate a menu class to add or modify menu items
* `generate:widgets` Let's you generate a widgets class to add or modify widgets
* `generate:tasks` Let's you generate a tasks class to add or modify tasks
* `development:enable` Let's you enable the development mode which will will disable some caching to make code changes directly visible and it will assist developers by performing additional checks to prevent for instance typos. Should not be used in production.
* `development:disable` Let's you disable the development mode
<!--
## Template: Piwik version number
### Breaking Changes
### Deprecations
### New features
### New APIs
### New commands
### New guides
### Library updates
### Internal change
-->
Find the general Piwik Changelogs for each release at [piwik.org/changelog](http://piwik.org/changelog/)

12
www/analytics/CONTRIBUTING.md Normal file
View file

@ -0,0 +1,12 @@
# How to contribute
Great to have you here! Read the following guide on our developer zone to learn how you can help make this project better!
http://developer.piwik.org/guides/contributing-to-piwik-core
## How to submit a bug report or suggest a feature?
Please read the recommendations on writing a good [bug report](http://developer.piwik.org/guides/core-team-workflow#submitting-a-bug-report) or [feature request](http://developer.piwik.org/guides/core-team-workflow#submitting-a-feature-request).
## How to suggest improvements to translations?
You can help improve translations in Piwik, please read [contribute to translations](https://github.com/piwik/piwik/blob/master/lang/README.md).

71
www/analytics/LEGALNOTICE
View file

@ -1,10 +1,10 @@
COPYRIGHT
Piwik - Open Source Web Analytics
Piwik - free/libre analytics platform
The software package is:
Copyright (C) 2013 Matthieu Aubry
Copyright (C) 2014 Matthieu Aubry
Individual contributions, components, and libraries are copyright
of their respective authors.
@ -40,7 +40,7 @@ CREDITS
For detailed contribution history, refer to the source, tickets,
patches, and Git revision history, available at
http://dev.piwik.org/trac/
https://github.com/piwik/piwik/issues
https://github.com/piwik/piwik
@ -65,9 +65,17 @@ SEPARATELY LICENSED COMPONENTS AND LIBRARIES
Link: https://github.com/piwik/piwik/blob/master/libs/PiwikTracker/
License: New BSD
Name: UserAgentParser
Link: https://github.com/piwik/piwik/blob/master/libs/UserAgentParser/
License: New BSD
Name: DeviceDetector
Link: https://github.com/piwik/device-detector
License: LGPL
Name: Piwik/Decompress
Link: https://github.com/piwik/component-decompress
License: LGPL v3.0
Name: Piwik/Network
Link: https://github.com/piwik/component-network
License: LGPL v3.0
THIRD-PARTY COMPONENTS AND LIBRARIES
@ -77,40 +85,40 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: jqPlot
Link: http://www.jqplot.com/
License: Dual-licensed: MIT or GPL v2
License: Dual-licensed: MIT (Expat) or GPL v2
Name: jQuery
Link: http://jquery.com/
License: Dual-licensed: MIT or GPL
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
- includes Sizzle.js - multi-licensed: MIT, New BSD, or GPL [v2]
- includes Sizzle.js - multi-licensed: MIT (Expat), New BSD, or GPL [v2]
Name: jQuery UI
Link: http://jqueryui.com/
License: Dual-licensed: MIT or GPL
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
Name: jquery.history
Link: http://tkyk.github.com/jquery-history-plugin/
License: MIT
License: MIT (Expat)
Name: jquery.scrollTo
Link: http://plugins.jquery.com/project/ScrollTo
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery Tooltip
Link: http://bassistance.de/jquery-plugins/jquery-plugin-tooltip/
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery placeholder
Link: http://mths.be/placeholder
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery smartbanner
Link: https://github.com/jasny/jquery.smartbanner
License: Dual licensed: MIT
License: Dual licensed: MIT (Expat)
Name: json2.js
Link: http://json.org/
@ -187,8 +195,8 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: Zend Framework
Link: http://www.zendframework.com/
License: New BSD
Name: pChart 2.1.3
Name: pChart 2.1.4
Link: http://www.pchart.net
License: GPL v3
@ -206,15 +214,27 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: Raphaël - JavaScript Vector Library
Link: http://raphaeljs.com/
License: MIT
License: MIT (Expat)
Name: lessphp
Link: http://leafo.net/lessphp
License: GPL3/MIT
License: GPL3, MIT (Expat)
Name: Symfony Console Component
Link: https://github.com/symfony/Console
License: MIT
License: MIT (Expat)
Name: AngularJS
Link: https://github.com/angular/angular.js
License: MIT (Expat)
Name: Mousetrap
Link: https://github.com/ccampbell/mousetrap
License: Apache 2.0
Name: PHP-DI
Link: http://php-di.org/
License: MIT (Expat)
THIRD-PARTY CONTENT
@ -235,10 +255,6 @@ THIRD-PARTY CONTENT
Notes:
- used in ImageGraph plugin
Name: plugins/CorePluginsAdmin/images/themes.png
Link: https://www.iconfinder.com/icons/17022/colors_draw_paint_icon
License: Free for commercial use
Name: plugins/Feedback/angularjs/ratefeature/thumbs-down.png
Link: https://www.iconfinder.com/icons/216428/down_thumbs_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
@ -247,10 +263,6 @@ THIRD-PARTY CONTENT
Link: https://www.iconfinder.com/icons/216429/thumbs_up_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
Name: plugins/CorePluginsAdmin/images/plugins.png
Link: http://findicons.com/icon/94051/tools_wizard?id=396912
License: GNU/GPL
Name: plugins/Insights/images/idea.png
Link: https://www.iconfinder.com/icons/6074/brainstorm_bulb_idea_jabber_light_icon
License: GPL
@ -259,7 +271,6 @@ THIRD-PARTY CONTENT
Notes:
- the "New BSD" license refers to either the "Modified BSD" and "Simplified BSD"
licenses (2- or 3-clause), which are GPL compatible.
- the "MIT" license is also referred to as the "X11" license
- icons for browsers, operating systems, browser plugins, search engines, and
and flags of countries are nominative use of third-party trademarks when
referring to the corresponding product or entity

60
www/analytics/PRIVACY.md Normal file
View file

@ -0,0 +1,60 @@
# Privacy
This is a summary of all of the components within Piwik which may affect your privacy in some way. Please keep in mind
third party Themes, Plugins or Apps may introduce privacy concerns not listed here.
## Privacy for users being tracked by Piwik
In this section we document how to protect the privacy of visitors who are tracked by your Piwik analytics service.
### Anonymise visitor IP addresses
By default, Piwik stores the visitor IP address (IPv4 or IPv6 format) in the database for each new visitor.
If a visitor has a static IP address this means her browsing history can be easily identified across several days and
even across several websites tracked within the same Piwik server. You can anonymize IP addresses to ensure visitors cannot
be tracked this way: [How to anonymise IP addresses.](http://piwik.org/docs/privacy/#step-1-automatically-anonymize-visitor-ips)
### Delete old visitors logs
By default, Piwik stores tracked data forever. To better respect the privacy of your users, it is recommended to regularly
purge old data. You can configure Piwik to automatically delete log data older than a specified number of months:
[How to delete old visitors log data.](http://piwik.org/docs/privacy/#step-2-delete-old-visitors-logs)
### Include a tracking Opt-Out feature on your site
In your website, we recommended providing an easy way for your visitors to “opt-out” of being tracked by Piwik.
You can use the Opt-Out feature to display a link your website that sets a special browser cookie (`piwik_ignore`) when
clicked. Visitors that click that link will be ignored by Piwik in the future:
[How to include a tracking opt-out iframe.](http://piwik.org/docs/privacy/#step-3-include-a-web-analytics-opt-out-feature-on-your-site-using-an-iframe)
### Respect DoNotTrack preference
Do Not Track is a browser-level technology and policy proposal that lets visitors opt out of tracking by websites they
do not visit. Visitors can enable this preference in their browser, and then it's up to Piwik to respect it. By default,
Piwik is configured to ignore visitors that have enabled it:
[How to check if your Piwik respects DoNotTrack.] (http://piwik.org/docs/privacy/#step-4-respect-donottrack-preference)
### Disable tracking cookies
A cookie is a collection of information that a website stores on a visitor’s computer and accesses each time the visitor
returns. By default, Piwik uses cookies to aid in tracking visitor behavior. If someone gains access to a visitor's
computer, they could learn a few things about how the visitor visited your website. For many websites, this isn't a
problem, but for others where a strong level of privacy is required (like online banking), disabling tracking cookies may
be a good idea: [How to disable tracking cookies.](http://piwik.org/faq/general/faq_157/)
### Keep your visitors details private
Any user that has at least `view` access (the default access level) to Piwik can view detailed information for all users
tracked in Piwik (such as their IP addresses, visitor IDs, details of all past visits and actions, etc.) through features
provided by the `Live` plugin (such as the Visitor Log and Visitor Profile). As the Piwik administrator, you may decide
that not all of your users need access to this data. You can deactivate the `Live` plugin to prevent users from viewing
visitor details in the Administration > Plugins page.
## Privacy for Piwik admins and website owners
In this section we document how a Piwik administrator can better protect their own privacy.
### Keep your Piwik server URL private
By default, the Piwik Javascript code on all tracked websites contains the Piwik server URL. In some cases you might
want to hide this Piwik URL completely while still tracking all websites in your Piwik instance. To hide your Piwik
server's URL, you can modify the Javascript Tracking code and point it to a proxy piwik.php script instead of your actual
Piwik server: [How to keep Piwik server URL private.](http://piwik.org/faq/how-to/faq_132/)
### Automatic update check
From time to time, Piwik uses `api.piwik.org` to check if the current version of Piwik is the latest version of Piwik.
If an update is available, a notification is displayed allowing you to upgrade Piwik. To disable the update check,
and stop your instance from sending HTTP requests to `api.piwik.org`, deactivate the "Automatic update" feature by
setting `enable_auto_update = 0` in your configuration file `config/config.ini.php`.
Learn more about [Privacy in Piwik](http://piwik.org/privacy/).

50
www/analytics/README.md
View file

@ -1,14 +1,27 @@
# Piwik - piwik.org
# Piwik - piwik.org
[![Latest Stable Version](https://poser.pugx.org/piwik/piwik/v/stable)](https://packagist.org/packages/piwik/piwik)
[![Latest Unstable Version](https://poser.pugx.org/piwik/piwik/v/unstable)](https://packagist.org/packages/piwik/piwik)
[![Total Downloads](https://poser.pugx.org/piwik/piwik/downloads)](https://packagist.org/packages/piwik/piwik)
[![License](https://poser.pugx.org/piwik/piwik/license)](https://packagist.org/packages/piwik/piwik)
## Code Status
[![Build Status](https://travis-ci.org/piwik/piwik.svg?branch=master)](https://travis-ci.org/piwik/piwik)
[![Scrutinizer Code Quality](https://img.shields.io/scrutinizer/g/piwik/piwik.svg)](https://scrutinizer-ci.com/g/piwik/piwik?branch=master)
[![Code Coverage](https://scrutinizer-ci.com/g/piwik/piwik/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/piwik/piwik/?branch=master)
[![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/piwik/piwik.svg)](http://isitmaintained.com/project/piwik/piwik "Average time to resolve an issue")
[![Percentage of issues still open](http://isitmaintained.com/badge/open/piwik/piwik.svg)](http://isitmaintained.com/project/piwik/piwik "Percentage of issues still open")
## Description
Piwik is the leading Free/Libre open source Web Analytics platform.
Piwik is the leading Free/Libre open analytics platform.
Piwik is a full featured PHP MySQL software program that you download and install on your own webserver.
At the end of the five minute installation process you will be given a JavaScript code.
Simply copy and paste this tag on websites you wish to track and access your analytics reports in real time.
Piwik aims to be a Free software alternative to Google Analytics, and is already used on more than 1,000,000 websites.
Piwik aims to be a Free software alternative to Google Analytics, and is already used on more than 1,000,000 websites. Privacy is built-in!
## Mission Statement
@ -21,17 +34,27 @@ Or in short:
Piwik is released under the GPL v3 (or later) license, see [misc/gpl-3.0.txt](misc/gpl-3.0.txt)
## We’re seeking a talented Software Engineer
Are you looking for a new challenge? We are currently seeking a software engineer or software developer who is passionate about data processing, security, privacy, the open source and free/libre philosophy and usable interface design.
[View Job Description](https://piwik.org/blog/2015/01/piwik-expanding-seeking-talented-software-engineer-new-zealand-poland/) - [Apply online](http://piwik.org/jobs/)
This is for a full time position to work on the open source Piwik platform, either remotely or we can help the right candidate relocate to beautiful New Zealand (Wellington) or Poland (Wroclaw).
We are grateful if you can share the job description with your friends and wider network!
## Requirements
* PHP 5.3.2 or greater
* PHP 5.3.3 or greater
* MySQL 4.1 or greater, and either MySQLi or PDO library must be enabled
* Piwik is OS / server independent
See http://piwik.org/docs/requirements/
## Install
## Install
* Upload piwik to your webserver
* Upload piwik to your webserver
* Point your browser to the directory
* Follow the steps
* Add the given javascript code to your pages
@ -43,7 +66,7 @@ If you do not have a server, consider our Piwik Hosting partner: http://piwik.or
## Changelog
For the list of all tickets closed in the current and past releases, see http://piwik.org/changelog/
For the list of all tickets closed in the current and past releases, see http://piwik.org/changelog/. For the list of technical changes in the Piwik platform, see [http://developer.piwik.org/changelog](http://developer.piwik.org/changelog).
## Participate!
@ -64,7 +87,7 @@ About us: http://piwik.org/the-piwik-team/
What makes Piwik unique from the competition:
* Real time web analytics reports: in Piwik, reports are by default generated in real time.
* Real time web analytics reports: in Piwik, reports are by default generated in real time.
For high traffic websites, you can choose the frequency for reports to be processed.
* You own your web analytics data: since Piwik is installed on your server, the data is stored in your own database and you can get all the statistics
@ -74,19 +97,12 @@ What makes Piwik unique from the competition:
* Modern, easy to use User Interface: you can fully customize your dashboard, drag and drop widgets and more.
* Piwik features are built inside plugins: you can add new features and remove the ones you don’t need.
* Piwik features are built inside plugins: you can add new features and remove the ones you don’t need.
You can build your own web analytics plugins or hire a consultant to have your custom feature built in Piwik
* Vibrant international Open community of more than 200,000 active users (tracking even more websites!)
* Advanced Web Analytics capabilities such as Ecommerce Tracking, Goal tracking, Campaign tracking,
* Advanced Web Analytics capabilities such as Ecommerce Tracking, Goal tracking, Campaign tracking,
Custom Variables, Email Reports, Custom Segment Editor, Geo Location, Real time maps, and more!
Documentation and more info on http://piwik.org
## Code Status
The Piwik project uses an ever-expanding comprehensive set of thousands of unit tests and dozens of integration [tests](https://github.com/piwik/piwik/tree/master/tests),
running on the hosted distributed continuous integration platform Travis-CI.
Build status (master branch) [![Build Status](https://travis-ci.org/piwik/piwik.png?branch=master)](https://travis-ci.org/piwik/piwik) - Screenshot tests Build [![Build Status](https://travis-ci.org/piwik/piwik-ui-tests.png?branch=master)](https://travis-ci.org/piwik/piwik-ui-tests)

21
www/analytics/SECURITY.md Normal file
View file

@ -0,0 +1,21 @@
# Reporting Security Issues
## Security Bug Bounty Program
The Piwik Security Bug Bounty Program is designed to encourage security research in Piwik software and to reward those who help us create the safest web analytics platform. The bounty for valid critical security bugs is a **$555** (US) cash reward. The bounty for non-critical bugs is **$242** (US), paid via Paypal.
## Responsible disclosure by email
If you have found a security issue in Piwik please read [our security notes](http://piwik.org/security/) regarding responsible disclosures.
[Email your Report Vulnerability to the Piwik Security team](mailto:security@piwik.org?subject=Reporting%20Vulnerability%20in%20Piwik)
## Improve your Piwik Server Security
[Secure Piwik server](http://piwik.org/docs/how-to-secure-piwik/): follow these steps to keep your Piwik data safe.
## Security announcements
Please subscribe to [the Changelog](http://piwik.org/changelog/) ([rss feed](http://piwik.org/changelog/feed/)) to be notified of new releases (including security releases).

41
www/analytics/bower.json Normal file
View file

@ -0,0 +1,41 @@
{
"name": "Piwik",
"main": "piwik.js",
"homepage": "http://piwik.org",
"authors": [
"Piwik.org <hello@piwik.org>"
],
"description": "the leading free/libre analytics platform",
"private": true,
"keywords": [
"piwik",
"web",
"analytics"
],
"dependencies": {
"jquery-ui": "1.10.4",
"jquery": "~1.11.0",
"angular": "~1.2.0",
"angular-sanitize": "~1.2.0",
"angular-animate": "~1.2.0",
"angular-cookies": "~1.2.0",
"angular-mocks": "~1.2.0",
"ngDialog": "~0.2.0",
"html5shiv": "~3.7.0",
"mousetrap": "~1.4.0",
"sprintf": "~1.0.0",
"jScrollPane": "~2.0.0",
"jquery-mousewheel": "~3.1.12",
"jquery-placeholder": "~2.0.8",
"jQuery.dotdotdot": "~1.7.2",
"jquery.scrollTo": "~1.4.13",
"chroma-js": "~0.6.0",
"visibilityjs": "~1.2.1"
},
"license": "GPLv3 or later",
"ignore": [
"**/.*",
"node_modules",
"tests"
]
}

101
www/analytics/composer.json
View file

@ -1,7 +1,7 @@
{
"name": "piwik/piwik",
"type": "application",
"description": "Open Source Real Time Web Analytics Platform",
"description": "the leading free/libre analytics platform",
"keywords": ["piwik","web","analytics"],
"homepage": "http://piwik.org",
"license": "GPL-3.0+",
@ -14,17 +14,98 @@
],
"support": {
"forum": "http://forum.piwik.org/",
"issues": "http://dev.piwik.org/trac/roadmap",
"wiki": "http://dev.piwik.org/",
"issues": "https://github.com/piwik/piwik/issues",
"wiki": "https://github.com/piwik/piwik/wiki",
"source": "https://github.com/piwik/piwik"
},
"autoload": {
"psr-4": {
"Piwik\\Plugins\\": "plugins/",
"Piwik\\": "core/"
},
"psr-0": {
"Zend_": "libs/",
"HTML_": "libs/",
"PEAR_": "libs/",
"Archive_": "libs/"
}
},
"autoload-dev": {
"psr-4": {
"Piwik\\Tests\\": "tests/PHPUnit/"
}
},
"require": {
"php": ">=5.3.2",
"twig/twig": "1.*",
"leafo/lessphp": "~0.3",
"symfony/console": ">=v2.3.5",
"tedivm/jshrink": "v0.5.1",
"mustangostang/spyc": "0.5.*",
"piwik/device-detector": "*"
"php": ">=5.3.3",
"twig/twig": "~1.0",
"leafo/lessphp": "~0.5.0",
"symfony/console": "~2.6.0",
"tedivm/jshrink": "~0.5.1",
"mustangostang/spyc": "~0.5.0",
"piwik/device-detector": "~3.0",
"piwik/decompress": "~1.0",
"piwik/network": "~0.1.0",
"piwik/cache": "~0.2.5",
"piwik/ini": "^1.0.6",
"php-di/php-di": "5.0.0-beta1",
"psr/log": "~1.0",
"monolog/monolog": "~1.11",
"symfony/monolog-bridge": "~2.6.0",
"symfony/event-dispatcher": "~2.6.0",
"pear/pear_exception": "~1.0.0",
"piwik/referrer-spam-blacklist": "~1.0",
"piwik/searchengine-and-social-list": "~1.0",
"tecnickcom/tcpdf": "~6.0",
"piwik/piwik-php-tracker": "^1.0"
},
"require-dev": {
"aws/aws-sdk-php": "2.7.1",
"phpunit/phpunit": "~4.8",
"facebook/xhprof": "dev-master",
"phpseclib/phpseclib": "~0.3.8",
"symfony/var-dumper": "~2.6.0",
"symfony/yaml": "~2.6.0"
},
"repositories": [
{
"type": "package",
"package": {
"name": "facebook/xhprof",
"type": "library",
"description": "XHProf: A Hierarchical Profiler for PHP",
"keywords": ["profiling", "performance"],
"homepage": "http://pecl.php.net/package/xhprof",
"license": "Apache-2.0",
"version": "master",
"require": {
"php": ">=5.2.0"
},
"autoload": {
"files": [
"xhprof_lib/utils/xhprof_lib.php",
"xhprof_lib/utils/xhprof_runs.php"
]
},
"source": {
"type": "git",
"url": "https://github.com/phacility/xhprof",
"reference": "master"
}
}
}
],
"scripts": {
"pre-update-cmd": [
"Piwik\\Composer\\ScriptHandler::cleanXhprof"
],
"pre-install-cmd": [
"Piwik\\Composer\\ScriptHandler::cleanXhprof"
],
"post-update-cmd": [
"Piwik\\Composer\\ScriptHandler::buildXhprof"
],
"post-install-cmd": [
"Piwik\\Composer\\ScriptHandler::buildXhprof"
]
}
}

2541
www/analytics/composer.lock generated
View file

File diff suppressed because it is too large Load diff

17
www/analytics/config/.htaccess
View file

@ -1,13 +1,8 @@
<Files "*">
<IfModule mod_access.c>
Deny from all
</IfModule>
<IfModule !mod_access_compat>
<IfModule mod_authz_host.c>
Deny from all
</IfModule>
</IfModule>
<IfModule mod_access_compat>
Deny from all
</IfModule>
<IfVersion < 2.4>
Deny from all
</IfVersion>
<IfVersion >= 2.4>
Require all denied
</IfVersion>
</Files>

12
www/analytics/config/environment/dev.php Normal file
View file

@ -0,0 +1,12 @@
<?php
return array(
'Piwik\Cache\Backend' => DI\object('Piwik\Cache\Backend\ArrayCache'),
'Piwik\Translation\Loader\LoaderInterface' => DI\object('Piwik\Translation\Loader\LoaderCache')
->constructor(DI\get('Piwik\Translation\Loader\DevelopmentLoader')),
'Piwik\Translation\Loader\DevelopmentLoader' => DI\object()
->constructor(DI\get('Piwik\Translation\Loader\JsonFileLoader')),
);

97
www/analytics/config/environment/test.php Normal file
View file

@ -0,0 +1,97 @@
<?php
use Interop\Container\ContainerInterface;
use Piwik\Common;
use Piwik\Tests\Framework\Mock\FakeAccess;
use Piwik\Tests\Framework\Mock\TestConfig;
return array(
// Disable logging
'Psr\Log\LoggerInterface' => DI\object('Psr\Log\NullLogger'),
'Piwik\Cache\Backend' => function () {
return \Piwik\Cache::buildBackend('file');
},
'cache.eager.cache_id' => 'eagercache-test-',
// Disable loading core translations
'Piwik\Translation\Translator' => DI\decorate(function ($previous, ContainerInterface $c) {
$loadRealTranslations = $c->get('test.vars.loadRealTranslations');
if (!$loadRealTranslations) {
return new \Piwik\Translation\Translator($c->get('Piwik\Translation\Loader\LoaderInterface'), $directories = array());
} else {
return $previous;
}
}),
'Piwik\Config' => DI\decorate(function ($previous, ContainerInterface $c) {
$testingEnvironment = $c->get('Piwik\Tests\Framework\TestingEnvironmentVariables');
$dontUseTestConfig = $c->get('test.vars.dontUseTestConfig');
if (!$dontUseTestConfig) {
$settingsProvider = $c->get('Piwik\Application\Kernel\GlobalSettingsProvider');
return new TestConfig($settingsProvider, $testingEnvironment, $allowSave = false, $doSetTestEnvironment = true);
} else {
return $previous;
}
}),
'Piwik\Access' => DI\decorate(function ($previous, ContainerInterface $c) {
$testUseMockAuth = $c->get('test.vars.testUseMockAuth');
if ($testUseMockAuth) {
$idSitesAdmin = $c->get('test.vars.idSitesAdminAccess');
$access = new FakeAccess();
if (!empty($idSitesAdmin)) {
FakeAccess::$superUser = false;
FakeAccess::$idSitesAdmin = $idSitesAdmin;
FakeAccess::$identity = 'adminUserLogin';
} else {
FakeAccess::$superUser = true;
FakeAccess::$superUserLogin = 'superUserLogin';
}
return $access;
} else {
return $previous;
}
}),
'observers.global' => DI\add(array(
array('AssetManager.getStylesheetFiles', function (&$stylesheets) {
$useOverrideCss = \Piwik\Container\StaticContainer::get('test.vars.useOverrideCss');
if ($useOverrideCss) {
$stylesheets[] = 'tests/resources/screenshot-override/override.css';
}
}),
array('AssetManager.getJavaScriptFiles', function (&$jsFiles) {
$useOverrideJs = \Piwik\Container\StaticContainer::get('test.vars.useOverrideJs');
if ($useOverrideJs) {
$jsFiles[] = 'tests/resources/screenshot-override/override.js';
}
}),
array('Updater.checkForUpdates', function () {
try {
@\Piwik\Filesystem::deleteAllCacheOnUpdate();
} catch (Exception $ex) {
// pass
}
}),
array('Test.Mail.send', function (\Zend_Mail $mail) {
$outputFile = PIWIK_INCLUDE_PATH . '/tmp/' . Common::getRequestVar('module', '') . '.' . Common::getRequestVar('action', '') . '.mail.json';
$outputContent = str_replace("=\n", "", $mail->getBodyText($textOnly = true));
$outputContent = str_replace("=0A", "\n", $outputContent);
$outputContent = str_replace("=3D", "=", $outputContent);
$outputContents = array(
'from' => $mail->getFrom(),
'to' => $mail->getRecipients(),
'subject' => $mail->getSubject(),
'contents' => $outputContent
);
file_put_contents($outputFile, json_encode($outputContents));
}),
)),
);

62
www/analytics/config/environment/ui-test.php Normal file
View file

@ -0,0 +1,62 @@
<?php
use Piwik\Container\StaticContainer;
return array(
// UI tests will remove the port from all URLs to the test server. if a test
// requires the ports in UI tests (eg, Overlay), add the api/controller methods
// to one of these blacklists
'tests.ui.url_normalizer_blacklist.api' => array(),
'tests.ui.url_normalizer_blacklist.controller' => array(),
'Piwik\Config' => \DI\decorate(function (\Piwik\Config $config) {
$config->General['cors_domains'][] = '*';
$config->General['trusted_hosts'][] = $config->tests['http_host'];
$config->General['trusted_hosts'][] = $config->tests['http_host'] . ':' . $config->tests['port'];
return $config;
}),
'observers.global' => \DI\add(array(
// removes port from all URLs to the test Piwik server so UI tests will pass no matter
// what port is used
array('Request.dispatch.end', function (&$result) {
$request = $_GET + $_POST;
$apiblacklist = StaticContainer::get('tests.ui.url_normalizer_blacklist.api');
if (!empty($request['method'])
&& in_array($request['method'], $apiblacklist)
) {
return;
}
$controllerActionblacklist = StaticContainer::get('tests.ui.url_normalizer_blacklist.controller');
if (!empty($request['module'])
&& !empty($request['action'])
) {
$controllerAction = $request['module'] . '.' . $request['action'];
if (in_array($controllerAction, $controllerActionblacklist)) {
return;
}
}
$config = \Piwik\Config::getInstance();
$host = $config->tests['http_host'];
$port = $config->tests['port'];
if (!empty($port)) {
// remove the port from URLs if any so UI tests won't fail if the port isn't 80
$result = str_replace($host . ':' . $port, $host, $result);
}
// remove PIWIK_INCLUDE_PATH from result so tests don't change based on the machine used
$result = str_replace(realpath(PIWIK_INCLUDE_PATH), '', $result);
}),
array('Controller.ExampleRssWidget.rssPiwik.end', function (&$result, $parameters) {
$result = "";
}),
)),
);

322
www/analytics/config/global.ini.php
View file

@ -18,25 +18,46 @@ password =
dbname =
tables_prefix =
port = 3306
adapter = PDO_MYSQL
adapter = PDO\MYSQL
type = InnoDB
schema = Mysql
; if charset is set to utf8, Piwik will ensure that it is storing its data using UTF8 charset.
; it will add a sql query SET at each page view.
; Piwik should work correctly without this setting.
;charset = utf8
; Piwik should work correctly without this setting but we recommend to have a charset set.
charset = utf8
[database_tests]
host = localhost
username = root
username = "@USERNAME@"
password =
dbname = piwik_tests
tables_prefix = piwiktests_
port = 3306
adapter = PDO_MYSQL
adapter = PDO\MYSQL
type = InnoDB
schema = Mysql
charset = utf8
[tests]
; needed in order to run tests.
; if Piwik is available at http://localhost/dev/piwik/ replace @REQUEST_URI@ with /dev/piwik/
; note: the REQUEST_URI should not contain "plugins" or "tests" in the PATH
http_host = localhost
remote_addr = "127.0.0.1"
request_uri = "@REQUEST_URI@"
port =
; access key and secret as listed in AWS -> IAM -> Users
aws_accesskey = ""
aws_secret = ""
; key pair name as listed in AWS -> EC2 -> Key Pairs. Key name should be different per user.
aws_keyname = ""
; PEM file can be downloaded after creating a new key pair in AWS -> EC2 -> Key Pairs
aws_pem_file = "<path to pem file>"
aws_securitygroups[] = "default"
aws_region = "us-east-1"
aws_ami = "ami-ac24bac4"
aws_instance_type = "c3.large"
[log]
; possible values for log: screen, database, file
@ -44,20 +65,36 @@ log_writers[] = screen
; log level, everything logged w/ this level or one of greater severity
; will be logged. everything else will be ignored. possible values are:
; NONE, ERROR, WARN, INFO, DEBUG, VERBOSE
; ERROR, WARN, INFO, DEBUG
log_level = WARN
; if set to 1, only requests done in CLI mode (eg. the archive.php cron run) will be logged
; NOTE: log_only_when_debug_parameter will also be checked for
log_only_when_cli = 0
; if set to 1, only requests with "&debug" parameter will be logged
; NOTE: log_only_when_cli will also be checked for
log_only_when_debug_parameter = 0
; if configured to log in a file, log entries will be made to this file
logger_file_path = tmp/logs/piwik.log
[Cache]
; available backends are 'file', 'array', 'null', 'redis', 'chained'
; 'array' will cache data only during one request
; 'null' will not cache anything at all
; 'file' will cache on the filesystem
; 'redis' will cache on a Redis server, use this if you are running Piwik with multiple servers. Further configuration in [RedisCache] is needed
; 'chained' will chain multiple cache backends. Further configuration in [ChainedCache] is needed
backend = chained
[ChainedCache]
; The chained cache will always try to read from the fastest backend first (the first listed one) to avoid requesting
; the same cache entry from the slowest backend multiple times in one request.
backends[] = array
backends[] = file
[RedisCache]
; Redis server configuration.
host = "127.0.0.1"
port = 6379
timeout = 0.0
password = ""
database = 14
; In case you are using queued tracking: Make sure to configure a different database! Otherwise queued requests might
; be flushed
[Debug]
; if set to 1, the archiving process will always be triggered, even if the archive has already been computed
@ -72,40 +109,68 @@ always_archive_data_range = 0;
; NOTE: you must also set [log] log_writers[] = "screen" to enable the profiler to print on screen
enable_sql_profiler = 0
; if set to 1, a Piwik tracking code will be included in the Piwik UI footer and will track visits, pages, etc. to idsite = 1
; this is useful for Piwik developers as an easy way to create data in their local Piwik
track_visits_inside_piwik_ui = 0
; if set to 1, javascript files will be included individually and neither merged nor minified.
; this option must be set to 1 when adding, removing or modifying javascript files
disable_merged_assets = 0
; If set to 1, all requests to piwik.php will be forced to be 'new visitors'
tracker_always_new_visitor = 0
; Allow automatic upgrades to Beta or RC releases
allow_upgrades_to_beta = 0
; if set to 1, all SQL queries will be logged using the DEBUG log level
log_sql_queries = 0
[DebugTests]
; Set to 1 by default. If you set to 0, the standalone plugins (with their own git repositories)
; will not be loaded when executing tests.
enable_load_standalone_plugins_during_tests = 1
; When set to 1, standalone plugins (those with their own git repositories)
; will be loaded when executing tests.
enable_load_standalone_plugins_during_tests = 0
[Development]
; Enables the development mode where we avoid most caching to make sure code changes will be directly applied as
; some caches are only invalidated after an update otherwise. When enabled it'll also performs some validation checks.
; For instance if you register a method in a widget we will verify whether the method actually exists and is public.
; If not, we will show you a helpful warning to make it easy to find simple typos etc.
enabled = 0
; if set to 1, javascript files will be included individually and neither merged nor minified.
; this option must be set to 1 when adding, removing or modifying javascript files
; Note that for quick debugging, instead of using below setting, you can add `&disable_merged_assets=1` to the Piwik URL
disable_merged_assets = 0
[General]
; the following settings control whether Unique Visitors will be processed for different period types.
; the following settings control whether Unique Visitors `nb_uniq_visitors` and Unique users `nb_users` will be processed for different period types.
; year and range periods are disabled by default, to ensure optimal performance for high traffic Piwik instances
; if you set it to 1 and want the Unique Visitors to be re-processed for reports in the past, drop all piwik_archive_* tables
; it is recommended to always enable Unique Visitors processing for 'day' periods
; it is recommended to always enable Unique Visitors and Unique Users processing for 'day' periods
enable_processing_unique_visitors_day = 1
enable_processing_unique_visitors_week = 1
enable_processing_unique_visitors_month = 1
enable_processing_unique_visitors_year = 0
enable_processing_unique_visitors_range = 0
; controls whether Unique Visitors will be processed for groups of websites. these metrics describe the number
; of unique visitors across the entire set of websites, so if a visitor visited two websites in the group, she
; would still only be counted as one. only relevant when using plugins that group sites together
enable_processing_unique_visitors_multiple_sites = 0
; The list of periods that are available in the Piwik calendar
; Example use case: custom date range requests are processed in real time,
; so they may take a few minutes on very high traffic website: you may remove "range" below to disable this period
enabled_periods_UI = "day,week,month,year,range"
enabled_periods_API = "day,week,month,year,range"
; whether to enable subquery cache for Custom Segment archiving queries
enable_segments_subquery_cache = 0
; Any segment subquery that matches more than segments_subquery_cache_limit IDs will not be cached,
; and the original subquery executed instead.
segments_subquery_cache_limit = 100000
; TTL: Time to live for cache files, in seconds. Default to 60 minutes
segments_subquery_cache_ttl = 3600
; when set to 1, all requests to Piwik will return a maintenance message without connecting to the DB
; this is useful when upgrading using the shell command, to prevent other users from accessing the UI while Upgrade is in progress
maintenance_mode = 0
; Defines the release channel that shall be used. Currently available values are:
; "latest_stable", "latest_beta", "latest_2x_stable", "latest_2x_beta"
release_channel = "latest_stable"
; character used to automatically create categories in the Actions > Pages, Outlinks and Downloads reports
; for example a URL like "example.com/blog/development/first-post" will create
; the page first-post in the subcategory development which belongs to the blog category
@ -143,10 +208,36 @@ browser_archiving_disabled_enforce = 0
; By default, users can create Segments which are to be processed in Real-time.
; Setting this to 0 will force all newly created Custom Segments to be "Pre-processed (faster, requires archive.php cron)"
; This can be useful if you want to prevent users from adding much load on the server.
; Note: any existing Segment set to "processed in Real time", will still be set to Real-time.
; this will only affect custom segments added or modified after this setting is changed.
; Notes:
; * any existing Segment set to "processed in Real time", will still be set to Real-time.
; this will only affect custom segments added or modified after this setting is changed.
; * when set to 0 then any user with at least 'view' access will be able to create pre-processed segments.
enable_create_realtime_segments = 1
; Whether to enable the "Suggest values for segment" in the Segment Editor panel.
; Set this to 0 in case your Piwik database is very big, and suggested values may not appear in time
enable_segment_suggested_values = 1
; By default, any user with a "view" access for a website can create segment assigned to this website.
; Set this to "admin" or "superuser" to require that users should have at least this access to create new segments.
; Note: anonymous user (even if it has view access) is not allowed to create or edit segment.
; Possible values are "view", "admin", "superuser"
adding_segment_requires_access = "view"
; Whether it is allowed for users to add segments that affect all websites or not. If there are many websites
; this admin option can be used to prevent users from performing an action that will have a major impact
; on Piwik performance.
allow_adding_segments_for_all_websites = 1
; When archiving segments for the first time, this determines the oldest date that will be archived.
; This option can be used to avoid archiving (for isntance) the lastN years for every new segment.
; Valid option values include: "beginning_of_time" (start date of archiving will not be changed)
; "segment_last_edit_time" (start date of archiving will be the earliest last edit date found,
; if none is found, the created date is used)
; "segment_creation_time" (start date of archiving will be the creation date of the segment)
; lastN where N is an integer (eg "last10" to archive for 10 days before the segment creation date)
process_new_segments_from = "beginning_of_time"
; this action name is used when the URL ends with a slash /
; it is useful to have an actual string to write in the UI
action_default_name = index
@ -158,6 +249,11 @@ default_language = en
; default number of elements in the datatable
datatable_default_limit = 10
; Each datatable report has a Row Limit selector at the bottom right.
; By default you can select from 5 to 500 rows. You may customise the values below
; -1 will be displayed as 'all' and it will export all rows (filter_limit=-1)
datatable_row_limits = "5,10,25,50,100,250,500,-1"
; default number of rows returned in API responses
; this value is overwritten by the '# Rows to display' selector.
; if set to -1, a click on 'Export as' will export all rows independently of the current '# Rows to display'.
@ -175,21 +271,32 @@ default_day = yesterday
default_period = day
; Time in seconds after which an archive will be computed again. This setting is used only for today's statistics.
; Defaults to 10 seconds so that by default, Piwik provides real time reporting.
; This setting is overriden in the UI, under "General Settings".
; This setting is only used if it hasn't been overriden via the UI yet, or if enable_general_settings_admin=0
time_before_today_archive_considered_outdated = 10
time_before_today_archive_considered_outdated = 150
; This setting is overriden in the UI, under "General Settings".
; The default value is to allow browsers to trigger the Piwik archiving process.
; This setting is only used if it hasn't been overriden via the UI yet, or if enable_general_settings_admin=0
enable_browser_archiving_triggering = 1
; By default, Piwik will force archiving of range periods from browser requests, even if enable_browser_archiving_triggering
; is set to 0. This can sometimes create too much of a demand on system resources. Setting this option to 0 and setting
; enable_browser_archiving_triggering to 0 will make sure ranges are not archived on browser request. Since the cron
; archiver does not archive ranges, you must either disable ranges or make sure the ranges users' want to see will be
; processed somehow.
archiving_range_force_on_browser_request = 1
; By default Piwik runs OPTIMIZE TABLE SQL queries to free spaces after deleting some data.
; If your Piwik tracks millions of pages, the OPTIMIZE TABLE queries might run for hours (seen in "SHOW FULL PROCESSLIST \g")
; so you can disable these special queries here:
enable_sql_optimize_queries = 1
; By default Piwik is purging complete date range archives to free spaces after deleting some data.
; If you are pre-processing custom ranges using CLI task to make them easily available in UI,
; you can prevent this action from happening by setting this parameter to value bigger than 1
purge_date_range_archives_after_X_days = 1
; MySQL minimum required version
; note: timezone support added in 4.1.3
minimum_mysql_version = 4.1
@ -200,7 +307,7 @@ minimum_pgsql_version = 8.3
; Minimum adviced memory limit in php.ini file (see memory_limit value)
minimum_memory_limit = 128
; Minimum memory limit enforced when archived via misc/cron/archive.php
; Minimum memory limit enforced when archived via ./console core:archive
minimum_memory_limit_when_archiving = 768
; Piwik will check that usernames and password have a minimum length, and will check that characters are "allowed"
@ -217,14 +324,15 @@ session_save_handler = files
; If set to 1, Piwik will automatically redirect all http:// requests to https://
; If SSL / https is not correctly configured on the server, this will break Piwik
; If you set this to 1, and your SSL configuration breaks later on, you can always edit this back to 0
; If you set this to 1, and your SSL configuration breaks later on, you can always edit this back to 0
; it is recommended for security reasons to always use Piwik over https
force_ssl = 0
; login cookie name
login_cookie_name = piwik_auth
; login cookie expiration (14 days)
; By default, the auth cookie is set only for the duration of session.
; if "Remember me" is checked, the auth cookie will be valid for 14 days by default
login_cookie_expire = 1209600
; The path on the server in which the cookie will be available on.
@ -237,6 +345,12 @@ login_password_recovery_email_address = "password-recovery@{DOMAIN}"
; name that appears as a Sender in the password recovery email
login_password_recovery_email_name = Piwik
; email address that appears as a Repy-to in the password recovery email
; if specified, {DOMAIN} will be replaced by the current Piwik domain
login_password_recovery_replyto_email_address = "no-reply@{DOMAIN}"
; name that appears as a Reply-to in the password recovery email
login_password_recovery_replyto_email_name = "No-reply"
; By default when user logs out he is redirected to Piwik "homepage" usually the Login form.
; Uncomment the next line to set a URL to redirect the user to after he logs out of Piwik.
; login_logout_url = http://...
@ -255,10 +369,21 @@ language_cookie_name = piwik_lang
; standard email address displayed when sending emails
noreply_email_address = "noreply@{DOMAIN}"
; standard email name displayed when sending emails. If not set, a default name will be used.
noreply_email_name = ""
; feedback email address;
; when testing, use your own email address or "nobody"
feedback_email_address = "feedback@piwik.org"
; using to set reply_to in reports e-mail to login of report creator
scheduled_reports_replyto_is_user_email_and_alias = 0
; scheduled reports truncate limit
; the report will be rendered with the first 23 rows and will aggregate other rows in a summary row
; 23 rows table fits in one portrait page
scheduled_reports_truncate = 23
; during archiving, Piwik will limit the number of results recorded, for performance reasons
; maximum number of rows for any of the Referrers tables (keywords, search engines, campaigns, etc.)
datatable_archiving_maximum_rows_referrers = 1000
@ -347,9 +472,20 @@ enable_trusted_host_check = 1
;trusted_hosts[] = example.com
;trusted_hosts[] = stats.example.com
; The release server is an essential part of the Piwik infrastructure/ecosystem
; to provide the latest software version.
latest_version_url = http://builds.piwik.org/latest.zip
; List of Cross-origin resource sharing domains (eg domain or subdomain names) when generating absolute URLs.
; Described here: http://en.wikipedia.org/wiki/Cross-origin_resource_sharing
;
; Examples:
;cors_domains[] = http://example.com
;cors_domains[] = http://stats.example.com
;
; Or you may allow cross domain requests for all domains with:
;cors_domains[] = *
; If you use this Piwik instance over multiple hostnames, Piwik will need to know
; a unique instance_id for this instance, so that Piwik can serve the right custom logo and tmp/* assets,
; independently of the hostname Piwik is currently running under.
; instance_id = stats.example.com
; The API server is an essential part of the Piwik infrastructure/ecosystem to
; provide services to Piwik installations, e.g., getLatestVersion and
@ -362,6 +498,10 @@ api_service_url = http://api.piwik.org
; eg. $period=range&date=previous10 becomes $period=day&date=previous10. Use this setting to override the $period value.
graphs_default_period_to_plot_when_period_range = day
; When the ImageGraph plugin is activated, enabling this option causes the image graphs to show the evolution
; within the selected period instead of the evolution across the last n periods.
graphs_show_evolution_within_selected_period = 0
; The Overlay plugin shows the Top X following pages, Top X downloads and Top X outlinks which followed
; a view of the current page. The value X can be set here.
overlay_following_pages_limit = 300
@ -411,7 +551,29 @@ enable_auto_update = 1
; If set to 0 it also disables the "sent plugin update emails" feature in general and the related setting in the UI.
enable_update_communication = 1
; This controls whether the pivotBy query parameter can be used with any dimension or just subtable
; dimensions. If set to 1, it will fetch a report with a segment for each row of the table being pivoted.
; At present, this is very inefficient, so it is disabled by default.
pivot_by_filter_enable_fetch_by_segment = 0
; This controls the default maximum number of columns to display in a pivot table. Since a pivot table displays
; a table's rows as columns, the number of columns can become very large, which will affect webpage layouts.
; Set to -1 to specify no limit. Note: The pivotByColumnLimit query parameter can be used to override this default
; on a per-request basis;
pivot_by_filter_default_column_limit = 10
; If set to 0 it will disable Piwik Pro advertisements in some places. For example in the installation screen, the
; Piwik Pro Ad widget will be removed etc.
piwik_pro_ads_enabled = 1
[Tracker]
; Piwik uses "Privacy by default" model. When one of your users visit multiple of your websites tracked in this Piwik,
; Piwik will create for this user a fingerprint that will be different across the multiple websites.
; If you want to track unique users across websites (for example when using the InterSites plugin) you may set this setting to 1.
; Note: setting this to 0 increases your users' privacy.
enable_fingerprinting_across_websites = 0
; Piwik uses first party cookies by default. If set to 1,
; the visit ID cookie will be set on the Piwik server domain as well
; this is useful when you want to do cross websites analysis
@ -421,14 +583,14 @@ use_third_party_id_cookie = 0
; Once enabled (set to 1) messages will be logged to all loggers defined in "[log] log_writers" config.
debug = 0
; There is a feature in the Tracking API that lets you create new visit at any given time, for example if you know that a different user/customer is using
; the app then you would want to tell Piwik to create a new visit (even though both users are using the same browser/computer).
; To prevent abuse and easy creation of fake visits, this feature requires admin token_auth by default
; If you wish to use this feature using the Javascript tracker, you can set the setting new_visit_api_requires_admin=0, and in Javascript write:
; _paq.push(['appendToTrackingUrl', 'new_visit=1']);
new_visit_api_requires_admin = 1
; This option is an alternative to the debug option above. When set to 1, you can debug tracker request by adding
; a debug=1 query paramater in the URL. All other HTTP requests will not have debug enabled. For security reasons this
; option should be only enabled if really needed and only for a short time frame. Otherwise anyone can set debug=1 and
; see the log output as well.
debug_on_demand = 0
; This setting should only be set to 1 in an intranet setting, where most users have the same configuration (browsers, OS)
; This setting is described in this FAQ: http://piwik.org/faq/how-to/faq_175/
; Note: generally this should only be set to 1 in an intranet setting, where most users have the same configuration (browsers, OS)
; and the same IP. If left to 0 in this setting, all visitors will be counted as one single visitor.
trust_visitors_cookies = 0
@ -436,9 +598,9 @@ trust_visitors_cookies = 0
; This is used only if use_third_party_id_cookie = 1
cookie_name = _pk_uid
; by default, the Piwik tracking cookie expires in 2 years
; by default, the Piwik tracking cookie expires in 13 months (365 + 28 days)
; This is used only if use_third_party_id_cookie = 1
cookie_expire = 63072000
cookie_expire = 33955200;
; The path on the server in which the cookie will be available on.
; Defaults to empty. See spec in http://curl.haxx.se/rfc/cookie_spec.html
@ -466,7 +628,7 @@ default_time_one_page_visit = 0
; The mapping is defined in core/DataFiles/LanguageToCountry.php,
enable_language_to_country_guess = 1
; When the misc/cron/archive.php cron hasn't been setup, we still need to regularly run some maintenance tasks.
; When the `./console core:archive` cron hasn't been setup, we still need to regularly run some maintenance tasks.
; Visits to the Tracker will try to trigger Scheduled Tasks (eg. scheduled PDF/HTML reports by email).
; Scheduled tasks will only run if 'Enable Piwik Archiving from Browser' is enabled in the General Settings.
; Tasks run once every hour maximum, they might not run every hour if traffic is low.
@ -479,13 +641,27 @@ ignore_visits_cookie_name = piwik_ignore
; Comma separated list of variable names that will be read to define a Campaign name, for example CPC campaign
; Example: If a visitor first visits 'index.php?piwik_campaign=Adwords-CPC' then it will be counted as a campaign referrer named 'Adwords-CPC'
; Includes by default the GA style campaign parameters
campaign_var_name = "pk_campaign,piwik_campaign,utm_campaign,utm_source,utm_medium"
campaign_var_name = "pk_cpn,pk_campaign,piwik_campaign,utm_campaign,utm_source,utm_medium"
; Comma separated list of variable names that will be read to track a Campaign Keyword
; Example: If a visitor first visits 'index.php?piwik_campaign=Adwords-CPC&piwik_kwd=My killer keyword' ;
; then it will be counted as a campaign referrer named 'Adwords-CPC' with the keyword 'My killer keyword'
; Includes by default the GA style campaign keyword parameter utm_term
campaign_keyword_var_name = "pk_kwd,piwik_kwd,pk_keyword,utm_term"
campaign_keyword_var_name = "pk_kwd,pk_keyword,piwik_kwd,utm_term"
; if set to 1, actions that contain different campaign information from the visitor's ongoing visit will
; be treated as the start of a new visit. This will include situations when campaign information was absent before,
; but is present now.
create_new_visit_when_campaign_changes = 1
; if set to 1, actions that contain different website referrer information from the visitor's ongoing visit
; will be treated as the start of a new visit. This will include situations when website referrer information was
; absent before, but is present now.
create_new_visit_when_website_referrer_changes = 0
; ONLY CHANGE THIS VALUE WHEN YOU DO NOT USE PIWIK ARCHIVING, SINCE THIS COULD CAUSE PARTIALLY MISSING ARCHIVE DATA
; Whether to force a new visit at midnight for every visitor. Default 1.
create_new_visit_after_midnight = 1
; maximum length of a Page Title or a Page URL recorded in the log_action.name table
page_maximum_length = 1024;
@ -494,9 +670,16 @@ page_maximum_length = 1024;
; TTL: Time to live for cache files, in seconds. Default to 5 minutes.
tracker_cache_file_ttl = 300
; Whether Bulk tracking requests to the Tracking API requires the token_auth to be set.
bulk_requests_require_authentication = 0
; Whether Bulk tracking requests will be wrapped within a DB Transaction.
; This greatly increases performance of Log Analytics and in general any Bulk Tracking API requests.
bulk_requests_use_transaction = 1
; DO NOT USE THIS SETTING ON PUBLICLY AVAILABLE PIWIK SERVER
; !!! Security risk: if set to 0, it would allow anyone to push data to Piwik with custom dates in the past/future and even with fake IPs!
; When using the Tracking API, to override either the datetime and/or the visitor IP,
; When using the Tracking API, to override either the datetime and/or the visitor IP,
; token_auth with an "admin" access is required. If you set this setting to 0, the token_auth will not be required anymore.
; DO NOT USE THIS SETTING ON PUBLIC PIWIK SERVERS
tracking_requests_require_authentication = 1
@ -509,7 +692,7 @@ tracking_requests_require_authentication = 1
; for which all reports should be Archived during the cron execution
; All segment values MUST be URL encoded.
;Segments[]="visitorType==new"
;Segments[]="visitorType==returning"
;Segments[]="visitorType==returning,visitorType==returningCustomer"
; If you define Custom Variables for your visitor, for example set the visit type
;Segments[]="customVariableName1==VisitType;customVariableValue1==Customer"
@ -554,9 +737,12 @@ username = ; Proxy username: optional; if specified, password is mandatory
password = ; Proxy password: optional; if specified, username is mandatory
[Plugins]
; list of plugins (in order they will be loaded) that are activated by default in the Piwik platform
Plugins[] = CorePluginsAdmin
Plugins[] = CoreAdminHome
Plugins[] = CoreHome
Plugins[] = WebsiteMeasurable
Plugins[] = Diagnostics
Plugins[] = CoreVisualizations
Plugins[] = Proxy
Plugins[] = API
@ -568,8 +754,10 @@ Plugins[] = Actions
Plugins[] = Dashboard
Plugins[] = MultiSites
Plugins[] = Referrers
Plugins[] = UserSettings
Plugins[] = UserLanguage
Plugins[] = DevicesDetection
Plugins[] = Goals
Plugins[] = Ecommerce
Plugins[] = SEO
Plugins[] = Events
Plugins[] = UserCountry
@ -579,8 +767,8 @@ Plugins[] = VisitTime
Plugins[] = VisitorInterest
Plugins[] = ExampleAPI
Plugins[] = ExampleRssWidget
Plugins[] = Provider
Plugins[] = Feedback
Plugins[] = Monolog
Plugins[] = Login
Plugins[] = UsersManager
@ -599,29 +787,31 @@ Plugins[] = MobileMessaging
Plugins[] = Overlay
Plugins[] = SegmentEditor
Plugins[] = Insights
Plugins[] = Morpheus
Plugins[] = Contents
Plugins[] = BulkTracking
Plugins[] = Resolution
Plugins[] = DevicePlugins
Plugins[] = Heartbeat
Plugins[] = Intl
Plugins[] = PiwikPro
[PluginsInstalled]
PluginsInstalled[] = Diagnostics
PluginsInstalled[] = Login
PluginsInstalled[] = CoreAdminHome
PluginsInstalled[] = UsersManager
PluginsInstalled[] = SitesManager
PluginsInstalled[] = Installation
[Plugins_Tracker]
Plugins_Tracker[] = Provider
Plugins_Tracker[] = Goals
Plugins_Tracker[] = PrivacyManager
Plugins_Tracker[] = UserCountry
Plugins_Tracker[] = Login
PluginsInstalled[] = Monolog
PluginsInstalled[] = Intl
[APISettings]
; Any key/value pair can be added in this section, they will be available via the REST call
; index.php?module=API&method=API.getSettings
; index.php?module=API&method=API.getSettings
; This can be used to expose values from Piwik, to control for example a Mobile app tracking
SDK_batch_size = 10
SDK_interval_value = 30
; NOTE: do not directly edit this file! See notice at the top

85
www/analytics/config/global.php Normal file
View file

@ -0,0 +1,85 @@
<?php
use Interop\Container\ContainerInterface;
use Interop\Container\Exception\NotFoundException;
use Piwik\Cache\Eager;
use Piwik\SettingsServer;
return array(
'path.root' => PIWIK_USER_PATH,
'path.tmp' => function (ContainerInterface $c) {
$root = $c->get('path.root');
// TODO remove that special case and instead have plugins override 'path.tmp' to add the instance id
if ($c->has('ini.General.instance_id')) {
$instanceId = $c->get('ini.General.instance_id');
$instanceId = $instanceId ? '/' . $instanceId : '';
} else {
$instanceId = '';
}
return $root . '/tmp' . $instanceId;
},
'path.cache' => DI\string('{path.tmp}/cache/tracker/'),
'Piwik\Cache\Eager' => function (ContainerInterface $c) {
$backend = $c->get('Piwik\Cache\Backend');
$cacheId = $c->get('cache.eager.cache_id');
if (SettingsServer::isTrackerApiRequest()) {
$eventToPersist = 'Tracker.end';
$cacheId .= 'tracker';
} else {
$eventToPersist = 'Request.dispatch.end';
$cacheId .= 'ui';
}
$cache = new Eager($backend, $cacheId);
\Piwik\Piwik::addAction($eventToPersist, function () use ($cache) {
$cache->persistCacheIfNeeded(43200);
});
return $cache;
},
'Piwik\Cache\Backend' => function (ContainerInterface $c) {
try {
$backend = $c->get('ini.Cache.backend');
} catch (NotFoundException $ex) {
$backend = 'chained'; // happens if global.ini.php is not available
}
return \Piwik\Cache::buildBackend($backend);
},
'cache.eager.cache_id' => function () {
return 'eagercache-' . str_replace(array('.', '-'), '', \Piwik\Version::VERSION) . '-';
},
'Psr\Log\LoggerInterface' => DI\object('Psr\Log\NullLogger'),
'Piwik\Translation\Loader\LoaderInterface' => DI\object('Piwik\Translation\Loader\LoaderCache')
->constructor(DI\get('Piwik\Translation\Loader\JsonFileLoader')),
'observers.global' => array(),
'Piwik\EventDispatcher' => DI\object()->constructorParameter('observers', DI\get('observers.global')),
'Zend_Validate_EmailAddress' => function () {
return new \Zend_Validate_EmailAddress(array(
'hostname' => new \Zend_Validate_Hostname(array(
'tld' => false,
))));
},
'Piwik\Tracker\VisitorRecognizer' => DI\object()
->constructorParameter('trustCookiesOnly', DI\get('ini.Tracker.trust_visitors_cookies'))
->constructorParameter('visitStandardLength', DI\get('ini.Tracker.visit_standard_length'))
->constructorParameter('lookbackNSecondsCustom', DI\get('ini.Tracker.window_look_back_for_visitor'))
->constructorParameter('trackerAlwaysNewVisitor', DI\get('ini.Debug.tracker_always_new_visitor')),
'Piwik\Tracker\Settings' => DI\object()
->constructorParameter('isSameFingerprintsAcrossWebsites', DI\get('ini.Tracker.enable_fingerprinting_across_websites')),
);

7268
www/analytics/config/manifest.inc.php
View file

File diff suppressed because it is too large Load diff

25
www/analytics/console
View file

@ -3,26 +3,25 @@
if (!defined('PIWIK_DOCUMENT_ROOT')) {
define('PIWIK_DOCUMENT_ROOT', dirname(__FILE__) == '/' ? '' : dirname(__FILE__));
}
if (file_exists(PIWIK_DOCUMENT_ROOT . '/bootstrap.php')) {
require_once PIWIK_DOCUMENT_ROOT . '/bootstrap.php';
}
if (!defined('PIWIK_INCLUDE_PATH')) {
define('PIWIK_INCLUDE_PATH', PIWIK_DOCUMENT_ROOT);
}
if (!defined('PIWIK_USER_PATH')) {
define('PIWIK_USER_PATH', PIWIK_DOCUMENT_ROOT);
}
require_once PIWIK_INCLUDE_PATH . '/core/testMinimumPhpVersion.php';
require_once file_exists(PIWIK_INCLUDE_PATH . '/vendor/autoload.php')
? PIWIK_INCLUDE_PATH . '/vendor/autoload.php' // Piwik is the main project
: PIWIK_INCLUDE_PATH . '/../../autoload.php'; // Piwik is installed as a dependency
require_once PIWIK_INCLUDE_PATH . '/core/Loader.php';
require_once PIWIK_INCLUDE_PATH . '/libs/upgradephp/upgrade.php';
Piwik\Translate::loadEnglishTranslation();
require_once PIWIK_INCLUDE_PATH . '/core/bootstrap.php';
if (!Piwik\Common::isPhpCliMode()) {
exit;
}
if (!defined('PIWIK_ENABLE_ERROR_HANDLER') || PIWIK_ENABLE_ERROR_HANDLER) {
Piwik\ErrorHandler::registerErrorHandler();
Piwik\ExceptionHandler::setUp();
}
$console = new Piwik\Console();
$console->init();
$console->run();
$console->run();

17
www/analytics/core/.htaccess
View file

@ -1,13 +1,8 @@
<Files "*">
<IfModule mod_access.c>
Deny from all
</IfModule>
<IfModule !mod_access_compat>
<IfModule mod_authz_host.c>
Deny from all
</IfModule>
</IfModule>
<IfModule mod_access_compat>
Deny from all
</IfModule>
<IfVersion < 2.4>
Deny from all
</IfVersion>
<IfVersion >= 2.4>
Require all denied
</IfVersion>
</Files>

131
www/analytics/core/API/ApiRenderer.php Normal file
View file

@ -0,0 +1,131 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable\Renderer;
use Piwik\DataTable;
use Piwik\Piwik;
use Piwik\Plugin;
/**
* API renderer
*/
abstract class ApiRenderer
{
protected $request;
final public function __construct($request)
{
$this->request = $request;
$this->init();
}
protected function init()
{
}
abstract public function sendHeader();
public function renderSuccess($message)
{
return 'Success:' . $message;
}
public function renderException($message, \Exception $exception)
{
return $message;
}
public function renderScalar($scalar)
{
$dataTable = new DataTable\Simple();
$dataTable->addRowsFromArray(array($scalar));
return $this->renderDataTable($dataTable);
}
public function renderDataTable($dataTable)
{
$renderer = $this->buildDataTableRenderer($dataTable);
return $renderer->render();
}
public function renderArray($array)
{
$renderer = $this->buildDataTableRenderer($array);
return $renderer->render();
}
public function renderObject($object)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
public function renderResource($resource)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
/**
* @param $dataTable
* @return Renderer
*/
protected function buildDataTableRenderer($dataTable)
{
$format = self::getFormatFromClass(get_class($this));
if ($format == 'json2') {
$format = 'json';
}
$renderer = Renderer::factory($format);
$renderer->setTable($dataTable);
$renderer->setRenderSubTables(Common::getRequestVar('expanded', false, 'int', $this->request));
$renderer->setHideIdSubDatableFromResponse(Common::getRequestVar('hideIdSubDatable', false, 'int', $this->request));
return $renderer;
}
/**
* @param string $format
* @param array $request
* @return ApiRenderer
* @throws Exception
*/
public static function factory($format, $request)
{
$formatToCheck = '\\' . ucfirst(strtolower($format));
$rendererClassnames = Plugin\Manager::getInstance()->findMultipleComponents('Renderer', 'Piwik\\API\\ApiRenderer');
foreach ($rendererClassnames as $klassName) {
if (Common::stringEndsWith($klassName, $formatToCheck)) {
return new $klassName($request);
}
}
$availableRenderers = array();
foreach ($rendererClassnames as $rendererClassname) {
$availableRenderers[] = self::getFormatFromClass($rendererClassname);
}
$availableRenderers = implode(', ', $availableRenderers);
Common::sendHeader('Content-Type: text/plain; charset=utf-8');
throw new Exception(Piwik::translate('General_ExceptionInvalidRendererFormat', array($format, $availableRenderers)));
}
private static function getFormatFromClass($klassname)
{
$klass = explode('\\', $klassname);
return strtolower(end($klass));
}
}

41
www/analytics/core/API/CORSHandler.php Normal file
View file

@ -0,0 +1,41 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Piwik\Url;
class CORSHandler
{
/**
* @var array
*/
protected $domains;
public function __construct()
{
$this->domains = Url::getCorsHostsFromConfig();
}
public function handle()
{
// allow Piwik to serve data to all domains
if (in_array("*", $this->domains)) {
header('Access-Control-Allow-Origin: *');
return;
}
// specifically allow if it is one of the whitelisted CORS domains
if (!empty($_SERVER['HTTP_ORIGIN'])) {
$origin = $_SERVER['HTTP_ORIGIN'];
if (in_array($origin, $this->domains, true)) {
header('Access-Control-Allow-Origin: ' . $_SERVER['HTTP_ORIGIN']);
}
}
}
}

174
www/analytics/core/API/DataTableGenericFilter.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -11,20 +11,37 @@ namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\Filter\AddColumnsProcessedMetricsGoal;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
class DataTableGenericFilter
{
private static $genericFiltersInfo = null;
/**
* List of filter names not to run.
*
* @var string[]
*/
private $disabledFilters = array();
/**
* @var Report
*/
private $report;
/**
* @var array
*/
private $request;
/**
* Constructor
*
* @param $request
*/
function __construct($request)
public function __construct($request, $report)
{
$this->request = $request;
$this->report = $report;
}
/**
@ -37,6 +54,16 @@ class DataTableGenericFilter
$this->applyGenericFilters($table);
}
/**
* Makes sure a set of filters are not run.
*
* @param string[] $filterNames The name of each filter to disable.
*/
public function disableFilters($filterNames)
{
$this->disabledFilters = array_unique(array_merge($this->disabledFilters, $filterNames));
}
/**
* Returns an array containing the information of the generic Filter
* to be applied automatically to the data resulting from the API calls.
@ -51,43 +78,54 @@ class DataTableGenericFilter
*/
public static function getGenericFiltersInformation()
{
if (is_null(self::$genericFiltersInfo)) {
self::$genericFiltersInfo = array(
'Pattern' => array(
'filter_column' => array('string', 'label'),
'filter_pattern' => array('string'),
),
'PatternRecursive' => array(
'filter_column_recursive' => array('string', 'label'),
'filter_pattern_recursive' => array('string'),
),
'ExcludeLowPopulation' => array(
'filter_excludelowpop' => array('string'),
'filter_excludelowpop_value' => array('float', '0'),
),
'AddColumnsProcessedMetrics' => array(
'filter_add_columns_when_show_all_columns' => array('integer')
),
'AddColumnsProcessedMetricsGoal' => array(
'filter_update_columns_when_show_all_goals' => array('integer'),
'idGoal' => array('string', AddColumnsProcessedMetricsGoal::GOALS_OVERVIEW),
),
'Sort' => array(
'filter_sort_column' => array('string'),
'filter_sort_order' => array('string', 'desc'),
),
'Truncate' => array(
'filter_truncate' => array('integer'),
),
'Limit' => array(
'filter_offset' => array('integer', '0'),
'filter_limit' => array('integer'),
'keep_summary_row' => array('integer', '0'),
),
);
return array(
array('Pattern',
array(
'filter_column' => array('string', 'label'),
'filter_pattern' => array('string')
)),
array('PatternRecursive',
array(
'filter_column_recursive' => array('string', 'label'),
'filter_pattern_recursive' => array('string'),
)),
array('ExcludeLowPopulation',
array(
'filter_excludelowpop' => array('string'),
'filter_excludelowpop_value' => array('float', '0'),
)),
array('Sort',
array(
'filter_sort_column' => array('string'),
'filter_sort_order' => array('string', 'desc'),
)),
array('Truncate',
array(
'filter_truncate' => array('integer'),
)),
array('Limit',
array(
'filter_offset' => array('integer', '0'),
'filter_limit' => array('integer'),
'keep_summary_row' => array('integer', '0'),
))
);
}
private function getGenericFiltersHavingDefaultValues()
{
$filters = self::getGenericFiltersInformation();
if ($this->report && $this->report->getDefaultSortColumn()) {
foreach ($filters as $index => $filter) {
if ($filter[0] === 'Sort') {
$filters[$index][1]['filter_sort_column'] = array('string', $this->report->getDefaultSortColumn());
$filters[$index][1]['filter_sort_order'] = array('string', $this->report->getDefaultSortOrder());
}
}
}
return self::$genericFiltersInfo;
return $filters;
}
/**
@ -107,13 +145,20 @@ class DataTableGenericFilter
return;
}
$genericFilters = self::getGenericFiltersInformation();
$genericFilters = $this->getGenericFiltersHavingDefaultValues();
$filterApplied = false;
foreach ($genericFilters as $filterName => $parameters) {
foreach ($genericFilters as $filterMeta) {
$filterName = $filterMeta[0];
$filterParams = $filterMeta[1];
$filterParameters = array();
$exceptionRaised = false;
foreach ($parameters as $name => $info) {
if (in_array($filterName, $this->disabledFilters)) {
continue;
}
foreach ($filterParams as $name => $info) {
// parameter type to cast to
$type = $info[0];
@ -123,12 +168,6 @@ class DataTableGenericFilter
$defaultValue = $info[1];
}
// third element in the array, if it exists, overrides the name of the request variable
$varName = $name;
if (isset($info[2])) {
$varName = $info[2];
}
try {
$value = Common::getRequestVar($name, $defaultValue, $type, $this->request);
settype($value, $type);
@ -144,6 +183,45 @@ class DataTableGenericFilter
$filterApplied = true;
}
}
return $filterApplied;
}
public function areProcessedMetricsNeededFor($metrics)
{
$columnQueryParameters = array(
'filter_column',
'filter_column_recursive',
'filter_excludelowpop',
'filter_sort_column'
);
foreach ($columnQueryParameters as $queryParamName) {
$queryParamValue = Common::getRequestVar($queryParamName, false, $type = null, $this->request);
if (!empty($queryParamValue)
&& $this->containsProcessedMetric($metrics, $queryParamValue)
) {
return true;
}
}
return false;
}
/**
* @param ProcessedMetric[] $metrics
* @param string $name
* @return bool
*/
private function containsProcessedMetric($metrics, $name)
{
foreach ($metrics as $metric) {
if ($metric instanceof ProcessedMetric
&& $metric->getName() == $name
) {
return true;
}
}
return false;
}
}

44
www/analytics/core/API/DataTableManipulator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,7 +10,6 @@ namespace Piwik\API;
use Exception;
use Piwik\Archive\DataTableFactory;
use Piwik\Common;
use Piwik\DataTable\Row;
use Piwik\DataTable;
use Piwik\Period\Range;
@ -63,7 +62,7 @@ abstract class DataTableManipulator
{
if ($dataTable instanceof DataTable\Map) {
return $this->manipulateDataTableMap($dataTable);
} else if ($dataTable instanceof DataTable) {
} elseif ($dataTable instanceof DataTable) {
return $this->manipulateDataTable($dataTable);
} else {
return $dataTable;
@ -90,7 +89,7 @@ abstract class DataTableManipulator
* Manipulates a single DataTable instance. Derived classes must define
* this function.
*/
protected abstract function manipulateDataTable($dataTable);
abstract protected function manipulateDataTable($dataTable);
/**
* Load the subtable for a row.
@ -124,7 +123,7 @@ abstract class DataTableManipulator
}
}
$method = $this->getApiMethodForSubtable();
$method = $this->getApiMethodForSubtable($request);
return $this->callApiAndReturnDataTable($this->apiModule, $method, $request);
}
@ -136,19 +135,34 @@ abstract class DataTableManipulator
* @param $request
* @return
*/
protected abstract function manipulateSubtableRequest($request);
abstract protected function manipulateSubtableRequest($request);
/**
* Extract the API method for loading subtables from the meta data
*
* @throws Exception
* @return string
*/
private function getApiMethodForSubtable()
private function getApiMethodForSubtable($request)
{
if (!$this->apiMethodForSubtable) {
$meta = API::getInstance()->getMetadata('all', $this->apiModule, $this->apiMethod);
if (!empty($request['idSite'])) {
$idSite = $request['idSite'];
} else {
$idSite = 'all';
}
if(empty($meta)) {
$apiParameters = array();
if (!empty($request['idDimension'])) {
$apiParameters['idDimension'] = $request['idDimension'];
}
if (!empty($request['idGoal'])) {
$apiParameters['idGoal'] = $request['idGoal'];
}
$meta = API::getInstance()->getMetadata($idSite, $this->apiModule, $this->apiMethod, $apiParameters);
if (empty($meta)) {
throw new Exception(sprintf(
"The DataTable cannot be manipulated: Metadata for report %s.%s could not be found. You can define the metadata in a hook, see example at: http://developer.piwik.org/api-reference/events#apigetreportmetadata",
$this->apiModule, $this->apiMethod
@ -171,6 +185,8 @@ abstract class DataTableManipulator
$request = $this->manipulateSubtableRequest($request);
$request['serialize'] = 0;
$request['expanded'] = 0;
$request['format'] = 'original';
$request['format_metrics'] = 0;
// don't want to run recursive filters on the subtables as they are loaded,
// otherwise the result will be empty in places (or everywhere). instead we
@ -179,14 +195,8 @@ abstract class DataTableManipulator
$dataTable = Proxy::getInstance()->call($class, $method, $request);
$response = new ResponseBuilder($format = 'original', $request);
$dataTable = $response->getResponse($dataTable);
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $request) == 0) {
if (method_exists($dataTable, 'applyQueuedFilters')) {
$dataTable->applyQueuedFilters();
}
}
$response->disableSendHeader();
$dataTable = $response->getResponse($dataTable, $apiModule, $method);
return $dataTable;
}
}

44
www/analytics/core/API/DataTableManipulator/Flattener.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -38,17 +38,16 @@ class Flattener extends DataTableManipulator
* Separator for building recursive labels (or paths)
* @var string
*/
public $recursiveLabelSeparator = ' - ';
public $recursiveLabelSeparator = '';
/**
* @param DataTable $dataTable
* @param string $recursiveLabelSeparator
* @return DataTable|DataTable\Map
*/
public function flatten($dataTable)
public function flatten($dataTable, $recursiveLabelSeparator)
{
if ($this->apiModule == 'Actions' || $this->apiMethod == 'getWebsites') {
$this->recursiveLabelSeparator = '/';
}
$this->recursiveLabelSeparator = $recursiveLabelSeparator;
return $this->manipulate($dataTable);
}
@ -72,9 +71,10 @@ class Flattener extends DataTableManipulator
}
$newDataTable = $dataTable->getEmptyClone($keepFilters);
foreach ($dataTable->getRows() as $row) {
$this->flattenRow($row, $newDataTable);
foreach ($dataTable->getRows() as $rowId => $row) {
$this->flattenRow($row, $rowId, $newDataTable);
}
return $newDataTable;
}
@ -84,15 +84,21 @@ class Flattener extends DataTableManipulator
* @param string $labelPrefix
* @param bool $parentLogo
*/
private function flattenRow(Row $row, DataTable $dataTable,
private function flattenRow(Row $row, $rowId, DataTable $dataTable,
$labelPrefix = '', $parentLogo = false)
{
$label = $row->getColumn('label');
if ($label !== false) {
$label = trim($label);
if (substr($label, 0, 1) == '/' && $this->recursiveLabelSeparator == '/') {
$label = substr($label, 1);
if ($this->recursiveLabelSeparator == '/') {
if (substr($label, 0, 1) == '/') {
$label = substr($label, 1);
} elseif ($rowId === DataTable::ID_SUMMARY_ROW && $labelPrefix && $label != DataTable::LABEL_SUMMARY_ROW) {
$label = ' - ' . $label;
}
}
$label = $labelPrefix . $label;
$row->setColumn('label', $label);
}
@ -103,7 +109,16 @@ class Flattener extends DataTableManipulator
$row->setMetadata('logo', $logo);
}
$subTable = $this->loadSubtable($dataTable, $row);
/** @var DataTable $subTable */
$subTable = $row->getSubtable();
if ($subTable) {
$subTable->applyQueuedFilters();
$row->deleteMetadata('idsubdatatable_in_db');
} else {
$subTable = $this->loadSubtable($dataTable, $row);
}
$row->removeSubtable();
if ($subTable === null) {
@ -117,8 +132,8 @@ class Flattener extends DataTableManipulator
$dataTable->addRow($row);
}
$prefix = $label . $this->recursiveLabelSeparator;
foreach ($subTable->getRows() as $row) {
$this->flattenRow($row, $dataTable, $prefix, $logo);
foreach ($subTable->getRows() as $rowId => $row) {
$this->flattenRow($row, $rowId, $dataTable, $prefix, $logo);
}
}
}
@ -127,6 +142,7 @@ class Flattener extends DataTableManipulator
* Remove the flat parameter from the subtable request
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{

39
www/analytics/core/API/DataTableManipulator/LabelFilter.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -24,6 +24,7 @@ use Piwik\DataTable\Row;
class LabelFilter extends DataTableManipulator
{
const SEPARATOR_RECURSIVE_LABEL = '>';
const TERMINAL_OPERATOR = '@';
private $labels;
private $addLabelIndex;
@ -63,6 +64,10 @@ class LabelFilter extends DataTableManipulator
*/
private function doFilterRecursiveDescend($labelParts, $dataTable)
{
// we need to make sure to rebuild the index as some filters change the label column directly via
// $row->setColumn('label', '') which would not be noticed in the label index otherwise.
$dataTable->rebuildIndex();
// search for the first part of the tree search
$labelPart = array_shift($labelParts);
@ -101,6 +106,9 @@ class LabelFilter extends DataTableManipulator
protected function manipulateSubtableRequest($request)
{
unset($request['label']);
unset($request['flat']);
$request['totals'] = 0;
$request['filter_sort_column'] = ''; // do not sort, we only want to find a matching column
return $request;
}
@ -111,16 +119,22 @@ class LabelFilter extends DataTableManipulator
* Note: The HTML Encoded version must be tried first, since in ResponseBuilder the $label is unsanitized
* via Common::unsanitizeLabelParameter.
*
* @param string $label
* @param string $originalLabel
* @return array
*/
private function getLabelVariations($label)
private function getLabelVariations($originalLabel)
{
static $pageTitleReports = array('getPageTitles', 'getEntryPageTitles', 'getExitPageTitles');
$originalLabel = trim($originalLabel);
$isTerminal = substr($originalLabel, 0, 1) == self::TERMINAL_OPERATOR;
if ($isTerminal) {
$originalLabel = substr($originalLabel, 1);
}
$variations = array();
$label = urldecode($label);
$label = trim($label);
$label = trim(urldecode($originalLabel));
$sanitizedLabel = Common::sanitizeInputValue($label);
$variations[] = $sanitizedLabel;
@ -128,13 +142,20 @@ class LabelFilter extends DataTableManipulator
if ($this->apiModule == 'Actions'
&& in_array($this->apiMethod, $pageTitleReports)
) {
// special case: the Actions.getPageTitles report prefixes some labels with a blank.
// the blank might be passed by the user but is removed in Request::getRequestArrayFromString.
$variations[] = ' ' . $sanitizedLabel;
$variations[] = ' ' . $label;
if ($isTerminal) {
array_unshift($variations, ' ' . $sanitizedLabel);
array_unshift($variations, ' ' . $label);
} else {
// special case: the Actions.getPageTitles report prefixes some labels with a blank.
// the blank might be passed by the user but is removed in Request::getRequestArrayFromString.
$variations[] = ' ' . $sanitizedLabel;
$variations[] = ' ' . $label;
}
}
$variations[] = $label;
$variations = array_unique($variations);
return $variations;
}

189
www/analytics/core/API/DataTableManipulator/ReportTotalsCalculator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,13 +10,9 @@ namespace Piwik\API\DataTableManipulator;
use Piwik\API\DataTableManipulator;
use Piwik\DataTable;
use Piwik\DataTable\Row;
use Piwik\DataTable\BaseFilter;
use Piwik\Period\Range;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\Metrics;
use Piwik\Plugins\API\API;
use Piwik\Period;
use Piwik\Plugin\Report;
/**
* This class is responsible for setting the metadata property 'totals' on each dataTable if the report
@ -26,10 +22,29 @@ use Piwik\Plugins\API\API;
class ReportTotalsCalculator extends DataTableManipulator
{
/**
* Cached report metadata array.
* Array [readableMetric] => [summed value]
* @var array
*/
private static $reportMetadata = array();
private $totals = array();
/**
* @var Report
*/
private $report;
/**
* Constructor
*
* @param bool $apiModule
* @param bool $apiMethod
* @param array $request
* @param Report $report
*/
public function __construct($apiModule = false, $apiMethod = false, $request = array(), $report = null)
{
parent::__construct($apiModule, $apiMethod, $request);
$this->report = $report;
}
/**
* @param DataTable $table
@ -46,7 +61,7 @@ class ReportTotalsCalculator extends DataTableManipulator
try {
return $this->manipulate($table);
} catch(\Exception $e) {
} catch (\Exception $e) {
// eg. requests with idSubtable may trigger this exception
// (where idSubtable was removed in
// ?module=API&method=Events.getNameFromCategoryId&idSubtable=1&secondaryDimension=eventName&format=XML&idSite=1&period=day&date=yesterday&flat=0
@ -62,75 +77,32 @@ class ReportTotalsCalculator extends DataTableManipulator
*/
protected function manipulateDataTable($dataTable)
{
$report = $this->findCurrentReport();
if (!empty($report) && empty($report['dimension'])) {
if (!empty($this->report) && !$this->report->getDimension() && !$this->isAllMetricsReport()) {
// we currently do not calculate the total value for reports having no dimension
return $dataTable;
}
// Array [readableMetric] => [summed value]
$totalValues = array();
$this->totals = array();
$firstLevelTable = $this->makeSureToWorkOnFirstLevelDataTable($dataTable);
$metricsToCalculate = Metrics::getMetricIdsToProcessReportTotal();
$metricNames = array();
foreach ($metricsToCalculate as $metricId) {
if (!$this->hasDataTableMetric($firstLevelTable, $metricId)) {
continue;
}
$metricNames[$metricId] = Metrics::getReadableColumnName($metricId);
}
foreach ($firstLevelTable->getRows() as $row) {
$totalValues = $this->sumColumnValueToTotal($row, $metricId, $totalValues);
foreach ($firstLevelTable->getRows() as $row) {
$columns = $row->getColumns();
foreach ($metricNames as $metricId => $metricName) {
$this->sumColumnValueToTotal($columns, $metricId, $metricName);
}
}
$dataTable->setMetadata('totals', $totalValues);
$dataTable->setMetadata('totals', $this->totals);
return $dataTable;
}
private function hasDataTableMetric(DataTable $dataTable, $metricId)
{
$firstRow = $dataTable->getFirstRow();
if (empty($firstRow)) {
return false;
}
if (false === $this->getColumn($firstRow, $metricId)) {
return false;
}
return true;
}
/**
* Returns column from a given row.
* Will work with 2 types of datatable
* - raw datatables coming from the archive DB, which columns are int indexed
* - datatables processed resulting of API calls, which columns have human readable english names
*
* @param Row|array $row
* @param int $columnIdRaw see consts in Metrics::
* @return mixed Value of column, false if not found
*/
private function getColumn($row, $columnIdRaw)
{
$columnIdReadable = Metrics::getReadableColumnName($columnIdRaw);
if ($row instanceof Row) {
$raw = $row->getColumn($columnIdRaw);
if ($raw !== false) {
return $raw;
}
return $row->getColumn($columnIdReadable);
}
return false;
}
private function makeSureToWorkOnFirstLevelDataTable($table)
{
if (!array_key_exists('idSubtable', $this->request)) {
@ -144,8 +116,8 @@ class ReportTotalsCalculator extends DataTableManipulator
$module = $this->apiModule;
$action = $this->apiMethod;
} else {
$module = $firstLevelReport['module'];
$action = $firstLevelReport['action'];
$module = $firstLevelReport->getModule();
$action = $firstLevelReport->getAction();
}
$request = $this->request;
@ -164,33 +136,56 @@ class ReportTotalsCalculator extends DataTableManipulator
}
}
return $this->callApiAndReturnDataTable($module, $action, $request);
$table = $this->callApiAndReturnDataTable($module, $action, $request);
if ($table instanceof DataTable\Map) {
$table = $table->mergeChildren();
}
return $table;
}
private function sumColumnValueToTotal(Row $row, $metricId, $totalValues)
private function sumColumnValueToTotal($columns, $metricId, $metricName)
{
$value = $this->getColumn($row, $metricId);
if (false === $value) {
return $totalValues;
$value = false;
if (array_key_exists($metricId, $columns)) {
$value = $columns[$metricId];
}
$metricName = Metrics::getReadableColumnName($metricId);
if ($value === false) {
// we do not add $metricId to $possibleMetricNames for a small performance improvement since in most cases
// $metricId should be present in $columns so we avoid this foreach loop
$possibleMetricNames = array(
$metricName,
// TODO: this and below is a hack to get report totals to work correctly w/ MultiSites.getAll. can be corrected
// when all metrics are described by Metadata classes & internal naming quirks are handled by core system.
'Goal_' . $metricName,
'Actions_' . $metricName
);
foreach ($possibleMetricNames as $possibleMetricName) {
if (array_key_exists($possibleMetricName, $columns)) {
$value = $columns[$possibleMetricName];
break;
}
}
if (array_key_exists($metricName, $totalValues)) {
$totalValues[$metricName] += $value;
if ($value === false) {
return;
}
}
if (array_key_exists($metricName, $this->totals)) {
$this->totals[$metricName] += $value;
} else {
$totalValues[$metricName] = $value;
$this->totals[$metricName] = $value;
}
return $totalValues;
}
/**
* Make sure to get all rows of the first level table.
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{
@ -198,6 +193,7 @@ class ReportTotalsCalculator extends DataTableManipulator
$request['expanded'] = 0;
$request['filter_limit'] = -1;
$request['filter_offset'] = 0;
$request['filter_sort_column'] = '';
$parametersToRemove = array('flat');
@ -213,38 +209,21 @@ class ReportTotalsCalculator extends DataTableManipulator
return $request;
}
private function getReportMetadata()
{
if (!empty(static::$reportMetadata)) {
return static::$reportMetadata;
}
static::$reportMetadata = API::getInstance()->getReportMetadata();
return static::$reportMetadata;
}
private function findCurrentReport()
{
foreach ($this->getReportMetadata() as $report) {
if ($this->apiMethod == $report['action']
&& $this->apiModule == $report['module']) {
return $report;
}
}
}
private function findFirstLevelReport()
{
foreach ($this->getReportMetadata() as $report) {
if (!empty($report['actionToLoadSubTables'])
&& $this->apiMethod == $report['actionToLoadSubTables']
&& $this->apiModule == $report['module']
foreach (Report::getAllReports() as $report) {
$actionToLoadSubtables = $report->getActionToLoadSubTables();
if ($actionToLoadSubtables == $this->apiMethod
&& $this->apiModule == $report->getModule()
) {
return $report;
}
}
return null;
}
private function isAllMetricsReport()
{
return $this->report->getModule() == 'API' && $this->report->getAction() == 'get';
}
}

436
www/analytics/core/API/DataTablePostProcessor.php Normal file
View file

@ -0,0 +1,436 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
use Exception;
use Piwik\API\DataTableManipulator\Flattener;
use Piwik\API\DataTableManipulator\LabelFilter;
use Piwik\API\DataTableManipulator\ReportTotalsCalculator;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\PivotByDimension;
use Piwik\Metrics\Formatter;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
/**
* Processes DataTables that should be served through Piwik's APIs. This processing handles
* special query parameters and computes processed metrics. It does not included rendering to
* output formats (eg, 'xml').
*/
class DataTablePostProcessor
{
const PROCESSED_METRICS_COMPUTED_FLAG = 'processed_metrics_computed';
/**
* @var null|Report
*/
private $report;
/**
* @var string[]
*/
private $request;
/**
* @var string
*/
private $apiModule;
/**
* @var string
*/
private $apiMethod;
/**
* @var Inconsistencies
*/
private $apiInconsistencies;
/**
* @var Formatter
*/
private $formatter;
private $callbackBeforeGenericFilters;
private $callbackAfterGenericFilters;
/**
* Constructor.
*/
public function __construct($apiModule, $apiMethod, $request)
{
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
$this->setRequest($request);
$this->report = Report::factory($apiModule, $apiMethod);
$this->apiInconsistencies = new Inconsistencies();
$this->setFormatter(new Formatter());
}
public function setFormatter(Formatter $formatter)
{
$this->formatter = $formatter;
}
public function setRequest($request)
{
$this->request = $request;
}
public function setCallbackBeforeGenericFilters($callbackBeforeGenericFilters)
{
$this->callbackBeforeGenericFilters = $callbackBeforeGenericFilters;
}
public function setCallbackAfterGenericFilters($callbackAfterGenericFilters)
{
$this->callbackAfterGenericFilters = $callbackAfterGenericFilters;
}
/**
* Apply post-processing logic to a DataTable of a report for an API request.
*
* @param DataTableInterface $dataTable The data table to process.
* @return DataTableInterface A new data table.
*/
public function process(DataTableInterface $dataTable)
{
// TODO: when calculating metrics before hand, only calculate for needed metrics, not all. NOTE:
// this is non-trivial since it will require, eg, to make sure processed metrics aren't added
// after pivotBy is handled.
$dataTable = $this->applyPivotByFilter($dataTable);
$dataTable = $this->applyTotalsCalculator($dataTable);
$dataTable = $this->applyFlattener($dataTable);
if ($this->callbackBeforeGenericFilters) {
call_user_func($this->callbackBeforeGenericFilters, $dataTable);
}
$dataTable = $this->applyGenericFilters($dataTable);
$this->applyComputeProcessedMetrics($dataTable);
if ($this->callbackAfterGenericFilters) {
call_user_func($this->callbackAfterGenericFilters, $dataTable);
}
// we automatically safe decode all datatable labels (against xss)
$dataTable->queueFilter('SafeDecodeLabel');
$dataTable = $this->convertSegmentValueToSegment($dataTable);
$dataTable = $this->applyQueuedFilters($dataTable);
$dataTable = $this->applyRequestedColumnDeletion($dataTable);
$dataTable = $this->applyLabelFilter($dataTable);
$dataTable = $this->applyMetricsFormatting($dataTable);
return $dataTable;
}
private function convertSegmentValueToSegment(DataTableInterface $dataTable)
{
$dataTable->filter('AddSegmentBySegmentValue', array($this->report));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segmentValue'));
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyPivotByFilter(DataTableInterface $dataTable)
{
$pivotBy = Common::getRequestVar('pivotBy', false, 'string', $this->request);
if (!empty($pivotBy)) {
$this->applyComputeProcessedMetrics($dataTable);
$reportId = $this->apiModule . '.' . $this->apiMethod;
$pivotByColumn = Common::getRequestVar('pivotByColumn', false, 'string', $this->request);
$pivotByColumnLimit = Common::getRequestVar('pivotByColumnLimit', false, 'int', $this->request);
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segmentValue'));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segment'));
$dataTable->filter('PivotByDimension', array($reportId, $pivotBy, $pivotByColumn, $pivotByColumnLimit,
PivotByDimension::isSegmentFetchingEnabledInConfig()));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTable|DataTableInterface|DataTable\Map
*/
public function applyFlattener($dataTable)
{
if (Common::getRequestVar('flat', '0', 'string', $this->request) == '1') {
$flattener = new Flattener($this->apiModule, $this->apiMethod, $this->request);
if (Common::getRequestVar('include_aggregate_rows', '0', 'string', $this->request) == '1') {
$flattener->includeAggregateRows();
}
$recursiveLabelSeparator = ' - ';
if ($this->report) {
$recursiveLabelSeparator = $this->report->getRecursiveLabelSeparator();
}
$dataTable = $flattener->flatten($dataTable, $recursiveLabelSeparator);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyTotalsCalculator($dataTable)
{
if (1 == Common::getRequestVar('totals', '1', 'integer', $this->request)) {
$calculator = new ReportTotalsCalculator($this->apiModule, $this->apiMethod, $this->request, $this->report);
$dataTable = $calculator->calculate($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyGenericFilters($dataTable)
{
// if the flag disable_generic_filters is defined we skip the generic filters
if (0 == Common::getRequestVar('disable_generic_filters', '0', 'string', $this->request)) {
$this->applyProcessedMetricsGenericFilters($dataTable);
$genericFilter = new DataTableGenericFilter($this->request, $this->report);
$self = $this;
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($genericFilter, $report, $self) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
if ($genericFilter->areProcessedMetricsNeededFor($processedMetrics)) {
$self->computeProcessedMetrics($table);
}
});
$label = self::getLabelFromRequest($this->request);
if (!empty($label)) {
$genericFilter->disableFilters(array('Limit', 'Truncate'));
}
$genericFilter->filter($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyProcessedMetricsGenericFilters($dataTable)
{
$addNormalProcessedMetrics = null;
try {
$addNormalProcessedMetrics = Common::getRequestVar(
'filter_add_columns_when_show_all_columns', null, 'integer', $this->request);
} catch (Exception $ex) {
// ignore
}
if ($addNormalProcessedMetrics !== null) {
$dataTable->filter('AddColumnsProcessedMetrics', array($addNormalProcessedMetrics));
}
$addGoalProcessedMetrics = null;
try {
$addGoalProcessedMetrics = Common::getRequestVar(
'filter_update_columns_when_show_all_goals', null, 'integer', $this->request);
} catch (Exception $ex) {
// ignore
}
if ($addGoalProcessedMetrics !== null) {
$idGoal = Common::getRequestVar(
'idGoal', DataTable\Filter\AddColumnsProcessedMetricsGoal::GOALS_OVERVIEW, 'string', $this->request);
$dataTable->filter('AddColumnsProcessedMetricsGoal', array($ignore = true, $idGoal));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyQueuedFilters($dataTable)
{
// if the flag disable_queued_filters is defined we skip the filters that were queued
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $this->request) == 0) {
$dataTable->applyQueuedFilters();
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyRequestedColumnDeletion($dataTable)
{
// use the ColumnDelete filter if hideColumns/showColumns is provided (must be done
// after queued filters are run so processed metrics can be removed, too)
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
$showRawMetrics = Common::getRequestVar('showRawMetrics', 0, 'int', $this->request);
if (!empty($hideColumns)
|| !empty($showColumns)
) {
$dataTable->filter('ColumnDelete', array($hideColumns, $showColumns));
} else if ($showRawMetrics !== 1) {
$this->removeTemporaryMetrics($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
*/
public function removeTemporaryMetrics(DataTableInterface $dataTable)
{
$allColumns = !empty($this->report) ? $this->report->getAllMetrics() : array();
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($report, $allColumns) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
$allTemporaryMetrics = array();
foreach ($processedMetrics as $metric) {
$allTemporaryMetrics = array_merge($allTemporaryMetrics, $metric->getTemporaryMetrics());
}
if (!empty($allTemporaryMetrics)) {
$table->filter('ColumnDelete', array($allTemporaryMetrics));
}
});
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyLabelFilter($dataTable)
{
$label = self::getLabelFromRequest($this->request);
// apply label filter: only return rows matching the label parameter (more than one if more than one label)
if (!empty($label)) {
$addLabelIndex = Common::getRequestVar('labelFilterAddLabelIndex', 0, 'int', $this->request) == 1;
$filter = new LabelFilter($this->apiModule, $this->apiMethod, $this->request);
$dataTable = $filter->filter($label, $dataTable, $addLabelIndex);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyMetricsFormatting($dataTable)
{
$formatMetrics = Common::getRequestVar('format_metrics', 0, 'string', $this->request);
if ($formatMetrics == '0') {
return $dataTable;
}
// in Piwik 2.X & below, metrics are not formatted in API responses except for percents.
// this code implements this inconsistency
$onlyFormatPercents = $formatMetrics === 'bc';
$metricsToFormat = null;
if ($onlyFormatPercents) {
$metricsToFormat = $this->apiInconsistencies->getPercentMetricsToFormat();
}
$dataTable->filter(array($this->formatter, 'formatMetrics'), array($this->report, $metricsToFormat));
return $dataTable;
}
/**
* Returns the value for the label query parameter which can be either a string
* (ie, label=...) or array (ie, label[]=...).
*
* @param array $request
* @return array
*/
public static function getLabelFromRequest($request)
{
$label = Common::getRequestVar('label', array(), 'array', $request);
if (empty($label)) {
$label = Common::getRequestVar('label', '', 'string', $request);
if (!empty($label)) {
$label = array($label);
}
}
$label = self::unsanitizeLabelParameter($label);
return $label;
}
public static function unsanitizeLabelParameter($label)
{
// this is needed because Proxy uses Common::getRequestVar which in turn
// uses Common::sanitizeInputValue. This causes the > that separates recursive labels
// to become &gt; and we need to undo that here.
$label = str_replace( htmlentities('>'), '>', $label);
return $label;
}
public function computeProcessedMetrics(DataTable $dataTable)
{
if ($dataTable->getMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG)) {
return;
}
/** @var ProcessedMetric[] $processedMetrics */
$processedMetrics = Report::getProcessedMetricsForTable($dataTable, $this->report);
if (empty($processedMetrics)) {
return;
}
$dataTable->setMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG, true);
foreach ($processedMetrics as $name => $processedMetric) {
if (!$processedMetric->beforeCompute($this->report, $dataTable)) {
continue;
}
foreach ($dataTable->getRows() as $row) {
if ($row->getColumn($name) === false) { // only compute the metric if it has not been computed already
$computedValue = $processedMetric->compute($row);
if ($computedValue !== false) {
$row->addColumn($name, $computedValue);
}
$subtable = $row->getSubtable();
if (!empty($subtable)) {
$this->computeProcessedMetrics($subtable);
}
}
}
}
}
public function applyComputeProcessedMetrics(DataTableInterface $dataTable)
{
$dataTable->filter(array($this, 'computeProcessedMetrics'));
}
}

242
www/analytics/core/API/DocumentationGenerator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -12,10 +12,10 @@ use Exception;
use Piwik\Common;
use Piwik\Piwik;
use Piwik\Url;
use ReflectionClass;
class DocumentationGenerator
{
protected $modulesToHide = array('CoreAdminHome', 'DBStats');
protected $countPluginsLoaded = 0;
/**
@ -47,63 +47,152 @@ class DocumentationGenerator
if (!empty($prefixUrls)) {
$prefixUrls = 'http://demo.piwik.org/';
}
$str = $toc = '';
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$parametersToSet = array(
'idSite' => Common::getRequestVar('idSite', 1, 'int'),
'period' => Common::getRequestVar('period', 'day', 'string'),
'date' => Common::getRequestVar('date', 'today', 'string')
);
foreach (Proxy::getInstance()->getMetadata() as $class => $info) {
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
if (in_array($moduleName, $this->modulesToHide)) {
$rClass = new ReflectionClass($class);
if (!Piwik::hasUserSuperUserAccess() && $this->checkIfClassCommentContainsHideAnnotation($rClass)) {
continue;
}
$toc .= "<a href='#$moduleName'>$moduleName</a><br/>";
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
// we prefix all URLs with $prefixUrls
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss1 = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$exampleUrlRss2 = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last5', 'period' => 'week',) + $parametersToSet);
$lastNUrls = ", RSS of the last <a target=_blank href='$exampleUrlRss1&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target=_blank href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target=_blank href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target=_blank href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
}
$str .= '</small>';
$str .= "</div>\n";
$toDisplay = $this->prepareModulesAndMethods($info, $moduleName);
foreach ($toDisplay as $moduleName => $methods) {
$toc .= $this->prepareModuleToDisplay($moduleName);
$str .= $this->prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls);
}
$str .= '<div style="margin:15px;"><a href="#topApiRef">↑ Back to top</a></div>';
}
$str = "<h2 id='topApiRef' name='topApiRef'>Quick access to APIs</h2>
$toc
$str";
return $str;
}
public function prepareModuleToDisplay($moduleName)
{
return "<a href='#$moduleName'>$moduleName</a><br/>";
}
public function prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls)
{
$str = '';
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$info['__documentation'] = $this->checkDocumentation($info['__documentation']);
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($methods as $methodName) {
if (Proxy::getInstance()->isDeprecatedMethod($class, $methodName)) {
continue;
}
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
$str .= $this->addExamples($class, $methodName, $prefixUrls);
}
$str .= '</small>';
$str .= "</div>\n";
}
return $str;
}
public function prepareModulesAndMethods($info, $moduleName)
{
$toDisplay = array();
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
$toDisplay[$moduleName][] = $methodName;
}
return $toDisplay;
}
public function addExamples($class, $methodName, $prefixUrls)
{
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$parametersToSet = array(
'idSite' => Common::getRequestVar('idSite', 1, 'int'),
'period' => Common::getRequestVar('period', 'day', 'string'),
'date' => Common::getRequestVar('date', 'today', 'string')
);
$str = '';
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$lastNUrls = ", RSS of the last <a target='_blank' href='$exampleUrlRss&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target='_blank' href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target='_blank' href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target='_blank' href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
return $str;
}
/**
* Check if Class contains @hide
*
* @param ReflectionClass $rClass instance of ReflectionMethod
* @return bool
*/
public function checkIfClassCommentContainsHideAnnotation(ReflectionClass $rClass)
{
return false !== strstr($rClass->getDocComment(), '@hide');
}
/**
* Check if documentation contains @hide annotation and deletes it
*
* @param $moduleToCheck
* @return mixed
*/
public function checkDocumentation($moduleToCheck)
{
if (strpos($moduleToCheck, '@hide') == true) {
$moduleToCheck = str_replace(strtok(strstr($moduleToCheck, '@hide'), "\n"), "", $moduleToCheck);
}
return $moduleToCheck;
}
private function getInterfaceString($moduleName, $class, $info, $parametersToSet, $outputExampleUrls, $prefixUrls)
{
$str = '';
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
if (Proxy::getInstance()->isDeprecatedMethod($class, $methodName)) {
continue;
}
$str .= $this->getMethodString($moduleName, $class, $parametersToSet, $outputExampleUrls, $prefixUrls, $methodName, $str);
}
$str .= '<div style="margin:15px;"><a href="#topApiRef">↑ Back to top</a></div>';
return $str;
}
@ -136,6 +225,7 @@ class DocumentationGenerator
'ip' => '194.57.91.215',
'idSites' => '1,2',
'idAlert' => '1',
'seconds' => '3600',
// 'segmentName' => 'browserCode',
);
@ -169,8 +259,8 @@ class DocumentationGenerator
$aParameters = Proxy::getInstance()->getParametersList($class, $methodName);
// Kindly force some known generic parameters to appear in the final list
// the parameter 'format' can be set to all API methods (used in tests)
// the parameter 'hideIdSubDatable' is used for integration tests only
// the parameter 'serialize' sets php outputs human readable, used in integration tests and debug
// the parameter 'hideIdSubDatable' is used for system tests only
// the parameter 'serialize' sets php outputs human readable, used in system tests and debug
// the parameter 'language' sets the language for the response (eg. country names)
// the parameter 'flat' reduces a hierarchical table to a single level by concatenating labels
// the parameter 'include_aggregate_rows' can be set to include inner nodes in flat reports
@ -183,12 +273,25 @@ class DocumentationGenerator
$aParameters['label'] = false;
$aParameters['flat'] = false;
$aParameters['include_aggregate_rows'] = false;
$aParameters['filter_limit'] = false; //@review without adding this, I can not set filter_limit in $otherRequestParameters integration tests
$aParameters['filter_sort_column'] = false; //@review without adding this, I can not set filter_sort_column in $otherRequestParameters integration tests
$aParameters['filter_offset'] = false; //@review without adding this, I can not set filter_offset in $otherRequestParameters system tests
$aParameters['filter_limit'] = false; //@review without adding this, I can not set filter_limit in $otherRequestParameters system tests
$aParameters['filter_sort_column'] = false; //@review without adding this, I can not set filter_sort_column in $otherRequestParameters system tests
$aParameters['filter_sort_order'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_excludelowpop'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_excludelowpop_value'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_column_recursive'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_pattern_recursive'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_truncate'] = false;
$aParameters['hideColumns'] = false;
$aParameters['showColumns'] = false;
$aParameters['filter_pattern_recursive'] = false;
$aParameters['pivotBy'] = false;
$aParameters['pivotByColumn'] = false;
$aParameters['pivotByColumnLimit'] = false;
$aParameters['disable_queued_filters'] = false;
$aParameters['disable_generic_filters'] = false;
$aParameters['expanded'] = false;
$aParameters['idDimenson'] = false;
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
$aParameters = array_merge(array('module' => 'API', 'method' => $moduleName . '.' . $methodName), $aParameters);
@ -235,4 +338,43 @@ class DocumentationGenerator
$sParameters = implode(", ", $asParameters);
return "($sParameters)";
}
private function getMethodString($moduleName, $class, $parametersToSet, $outputExampleUrls, $prefixUrls, $methodName)
{
$str = '';
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
// we prefix all URLs with $prefixUrls
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$lastNUrls = ", RSS of the last <a target='_blank' href='$exampleUrlRss&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target='_blank' href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target='_blank' href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target='_blank' href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
}
$str .= '</small>';
$str .= "</div>\n";
return $str;
}
}

42
www/analytics/core/API/Inconsistencies.php Normal file
View file

@ -0,0 +1,42 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
/**
* Contains logic to replicate inconsistencies in Piwik's API. This class exists
* to provide a way to clean up existing Piwik code and behavior without breaking
* backwards compatibility immediately.
*
* Code that handles the case when the 'format_metrics' query parameter value is
* 'bc' should be removed as well. This code is in API\Request and DataTablePostProcessor.
*
* Should be removed before releasing Piwik 3.0.
*/
class Inconsistencies
{
/**
* In Piwik 2.X and below, the "raw" API would format percent values but no others.
* This method returns the list of percent metrics that were returned from the API
* formatted so we can maintain BC.
*
* Used by DataTablePostProcessor.
*/
public function getPercentMetricsToFormat()
{
return array(
'bounce_rate',
'conversion_rate',
'interaction_rate',
'exit_rate',
'bounce_rate_returning',
'nb_visits_percentage',
'/.*_evolution/',
'/goal_.*_conversion_rate/'
);
}
}

173
www/analytics/core/API/Proxy.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -24,7 +24,7 @@ use ReflectionMethod;
*
* It will also log the performance of API calls (time spent, parameter values, etc.) if logger available
*
* @method static \Piwik\API\Proxy getInstance()
* @method static Proxy getInstance()
*/
class Proxy extends Singleton
{
@ -37,10 +37,7 @@ class Proxy extends Singleton
// when a parameter doesn't have a default value we use this
private $noDefaultValue;
/**
* protected constructor
*/
protected function __construct()
public function __construct()
{
$this->noDefaultValue = new NoDefaultValue();
}
@ -78,12 +75,14 @@ class Proxy extends Singleton
$this->checkClassIsSingleton($className);
$rClass = new ReflectionClass($className);
foreach ($rClass->getMethods() as $method) {
$this->loadMethodMetadata($className, $method);
}
if (!$this->shouldHideAPIMethod($rClass->getDocComment())) {
foreach ($rClass->getMethods() as $method) {
$this->loadMethodMetadata($className, $method);
}
$this->setDocumentation($rClass, $className);
$this->alreadyRegistered[$className] = true;
$this->setDocumentation($rClass, $className);
$this->alreadyRegistered[$className] = true;
}
}
/**
@ -164,11 +163,11 @@ class Proxy extends Singleton
/**
* Triggered before an API request is dispatched.
*
*
* This event can be used to modify the arguments passed to one or more API methods.
*
*
* **Example**
*
*
* Piwik::addAction('API.Request.dispatch', function (&$parameters, $pluginName, $methodName) {
* if ($pluginName == 'Actions') {
* if ($methodName == 'getPageUrls') {
@ -178,7 +177,7 @@ class Proxy extends Singleton
* }
* }
* });
*
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
* @param string $pluginName The name of the plugin the API method belongs to.
* @param string $methodName The name of the API method that will be called.
@ -187,20 +186,20 @@ class Proxy extends Singleton
/**
* Triggered before an API request is dispatched.
*
*
* This event exists for convenience and is triggered directly after the {@hook API.Request.dispatch}
* event is triggered. It can be used to modify the arguments passed to a **single** API method.
*
*
* _Note: This is can be accomplished with the {@hook API.Request.dispatch} event as well, however
* event handlers for that event will have to do more work._
*
*
* **Example**
*
*
* Piwik::addAction('API.Actions.getPageUrls', function (&$parameters) {
* // force use of a single website. for some reason.
* $parameters['idSite'] = 1;
* });
*
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
*/
Piwik::postEvent(sprintf('API.%s.%s', $pluginName, $methodName), array(&$finalParameters));
@ -218,16 +217,16 @@ class Proxy extends Singleton
/**
* Triggered directly after an API request is dispatched.
*
*
* This event exists for convenience and is triggered immediately before the
* {@hook API.Request.dispatch.end} event. It can be used to modify the output of a **single**
* API method.
*
*
* _Note: This can be accomplished with the {@hook API.Request.dispatch.end} event as well,
* however event handlers for that event will have to do more work._
*
* **Example**
*
*
* // append (0 hits) to the end of row labels whose row has 0 hits
* Piwik::addAction('API.Actions.getPageUrls', function (&$returnValue, $info)) {
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
@ -238,13 +237,13 @@ class Proxy extends Singleton
* }
* }, null, array('nb_hits'));
* }
*
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* could be a {@link Piwik\DataTable DataTable}.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
@ -257,20 +256,20 @@ class Proxy extends Singleton
/**
* Triggered directly after an API request is dispatched.
*
*
* This event can be used to modify the output of any API method.
*
*
* **Example**
*
*
* // append (0 hits) to the end of row labels whose row has 0 hits for any report that has the 'nb_hits' metric
* Piwik::addAction('API.Actions.getPageUrls', function (&$returnValue, $info)) {
* Piwik::addAction('API.Actions.getPageUrls.end', function (&$returnValue, $info)) {
* // don't process non-DataTable reports and reports that don't have the nb_hits column
* if (!($returnValue instanceof DataTableInterface)
* || in_array('nb_hits', $returnValue->getColumns())
* ) {
* return;
* }
*
*
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
* if ($hits === 0) {
* return $label . " (0 hits)";
@ -279,12 +278,12 @@ class Proxy extends Singleton
* }
* }, null, array('nb_hits'));
* }
*
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
@ -323,6 +322,14 @@ class Proxy extends Singleton
return $this->metadataArray[$class][$name]['parameters'];
}
/**
* Check if given method name is deprecated or not.
*/
public function isDeprecatedMethod($class, $methodName)
{
return $this->metadataArray[$class][$methodName]['isDeprecated'];
}
/**
* Returns the 'moduleName' part of '\\Piwik\\Plugins\\moduleName\\API'
*
@ -378,7 +385,6 @@ class Proxy extends Singleton
$requestValue = Common::getRequestVar($name, null, null, $parametersRequest);
} else {
try {
if ($name == 'segment' && !empty($parametersRequest['segment'])) {
// segment parameter is an exception: we do not want to sanitize user input or it would break the segment encoding
$requestValue = ($parametersRequest['segment']);
@ -405,7 +411,7 @@ class Proxy extends Singleton
}
/**
* Includes the class API by looking up plugins/UserSettings/API.php
* Includes the class API by looking up plugins/xxx/API.php
*
* @param string $fileName api class name eg. "API"
* @throws Exception
@ -428,29 +434,27 @@ class Proxy extends Singleton
*/
private function loadMethodMetadata($class, $method)
{
if ($method->isPublic()
&& !$method->isConstructor()
&& $method->getName() != 'getInstance'
&& false === strstr($method->getDocComment(), '@deprecated')
&& (!$this->hideIgnoredFunctions || false === strstr($method->getDocComment(), '@ignore'))
) {
$name = $method->getName();
$parameters = $method->getParameters();
$aParameters = array();
foreach ($parameters as $parameter) {
$nameVariable = $parameter->getName();
$defaultValue = $this->noDefaultValue;
if ($parameter->isDefaultValueAvailable()) {
$defaultValue = $parameter->getDefaultValue();
}
$aParameters[$nameVariable] = $defaultValue;
}
$this->metadataArray[$class][$name]['parameters'] = $aParameters;
$this->metadataArray[$class][$name]['numberOfRequiredParameters'] = $method->getNumberOfRequiredParameters();
if (!$this->checkIfMethodIsAvailable($method)) {
return;
}
$name = $method->getName();
$parameters = $method->getParameters();
$docComment = $method->getDocComment();
$aParameters = array();
foreach ($parameters as $parameter) {
$nameVariable = $parameter->getName();
$defaultValue = $this->noDefaultValue;
if ($parameter->isDefaultValueAvailable()) {
$defaultValue = $parameter->getDefaultValue();
}
$aParameters[$nameVariable] = $defaultValue;
}
$this->metadataArray[$class][$name]['parameters'] = $aParameters;
$this->metadataArray[$class][$name]['numberOfRequiredParameters'] = $method->getNumberOfRequiredParameters();
$this->metadataArray[$class][$name]['isDeprecated'] = false !== strstr($docComment, '@deprecated');
}
/**
@ -468,15 +472,56 @@ class Proxy extends Singleton
}
/**
* Returns the number of required parameters (parameters without default values).
*
* @param string $class The class name
* @param string $name The method name
* @return int The number of required parameters
* @param $docComment
* @return bool
*/
private function getNumberOfRequiredParameters($class, $name)
public function shouldHideAPIMethod($docComment)
{
return $this->metadataArray[$class][$name]['numberOfRequiredParameters'];
$hideLine = strstr($docComment, '@hide');
if ($hideLine === false) {
return false;
}
$hideLine = trim($hideLine);
$hideLine .= ' ';
$token = trim(strtok($hideLine, " "), "\n");
$hide = false;
if (!empty($token)) {
/**
* This event exists for checking whether a Plugin API class or a Plugin API method tagged
* with a `@hideXYZ` should be hidden in the API listing.
*
* @param bool &$hide whether to hide APIs tagged with $token should be displayed.
*/
Piwik::postEvent(sprintf('API.DocumentationGenerator.%s', $token), array(&$hide));
}
return $hide;
}
/**
* @param ReflectionMethod $method
* @return bool
*/
protected function checkIfMethodIsAvailable(ReflectionMethod $method)
{
if (!$method->isPublic() || $method->isConstructor() || $method->getName() === 'getInstance') {
return false;
}
if ($this->hideIgnoredFunctions && false !== strstr($method->getDocComment(), '@ignore')) {
return false;
}
if ($this->shouldHideAPIMethod($method->getDocComment())) {
return false;
}
return true;
}
/**
@ -500,7 +545,7 @@ class Proxy extends Singleton
private function checkClassIsSingleton($className)
{
if (!method_exists($className, "getInstance")) {
throw new Exception("$className that provide an API must be Singleton and have a 'static public function getInstance()' method.");
throw new Exception("$className that provide an API must be Singleton and have a 'public static function getInstance()' method.");
}
}
}

274
www/analytics/core/API/Request.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -17,49 +17,49 @@ use Piwik\PluginDeactivatedException;
use Piwik\SettingsServer;
use Piwik\Url;
use Piwik\UrlHelper;
use Piwik\Log;
use Piwik\Plugin\Manager as PluginManager;
/**
* Dispatches API requests to the appropriate API method.
*
*
* The Request class is used throughout Piwik to call API methods. The difference
* between using Request and calling API methods directly is that Request
* will do more after calling the API including: applying generic filters, applying queued filters,
* and handling the **flat** and **label** query parameters.
*
*
* Additionally, the Request class will **forward current query parameters** to the request
* which is more convenient than calling {@link Piwik\Common::getRequestVar()} many times over.
*
*
* In most cases, using a Request object to query the API is the correct approach.
*
* ### Post-processing
*
*
* The return value of API methods undergo some extra processing before being returned by Request.
* To learn more about what happens to API results, read [this](/guides/piwiks-web-api#extra-report-processing).
*
* ### Output Formats
*
*
* The value returned by Request will be serialized to a certain format before being returned.
* To see the list of supported output formats, read [this](/guides/piwiks-web-api#output-formats).
*
*
* ### Examples
*
*
* **Basic Usage**
*
* $request = new Request('method=UserSettings.getWideScreen&idSite=1&date=yesterday&period=week'
*
* $request = new Request('method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week'
* . '&format=xml&filter_limit=5&filter_offset=0')
* $result = $request->process();
* echo $result;
*
*
* **Getting a unrendered DataTable**
*
*
* // use the convenience method 'processRequest'
* $dataTable = Request::processRequest('UserSettings.getWideScreen', array(
* $dataTable = Request::processRequest('UserLanguage.getLanguage', array(
* 'idSite' => 1,
* 'date' => 'yesterday',
* 'period' => 'week',
* 'filter_limit' => 5,
* 'filter_offset' => 0
*
*
* 'format' => 'original', // this is the important bit
* ));
* echo "This DataTable has " . $dataTable->getRowsCount() . " rows.";
@ -69,41 +69,46 @@ use Piwik\UrlHelper;
*/
class Request
{
protected $request = null;
private $request = null;
/**
* Converts the supplied request string into an array of query paramater name/value
* mappings. The current query parameters (everything in `$_GET` and `$_POST`) are
* forwarded to request array before it is returned.
*
* @param string|array $request The base request string or array, eg,
* `'module=UserSettings&action=getWidescreen'`.
* @param string|array|null $request The base request string or array, eg,
* `'module=UserLanguage&action=getLanguage'`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
* @return array
*/
static public function getRequestArrayFromString($request)
public static function getRequestArrayFromString($request, $defaultRequest = null)
{
$defaultRequest = $_GET + $_POST;
if ($defaultRequest === null) {
$defaultRequest = self::getDefaultRequest();
$requestRaw = self::getRequestParametersGET();
if (!empty($requestRaw['segment'])) {
$defaultRequest['segment'] = $requestRaw['segment'];
$requestRaw = self::getRequestParametersGET();
if (!empty($requestRaw['segment'])) {
$defaultRequest['segment'] = $requestRaw['segment'];
}
if (!isset($defaultRequest['format_metrics'])) {
$defaultRequest['format_metrics'] = 'bc';
}
}
$requestArray = $defaultRequest;
if (!is_null($request)) {
if (is_array($request)) {
$url = array();
foreach ($request as $key => $value) {
$url[] = $key . "=" . $value;
}
$request = implode("&", $url);
$requestParsed = $request;
} else {
$request = trim($request);
$request = str_replace(array("\n", "\t"), '', $request);
$requestParsed = UrlHelper::getArrayFromQueryString($request);
}
$request = trim($request);
$request = str_replace(array("\n", "\t"), '', $request);
$requestParsed = UrlHelper::getArrayFromQueryString($request);
$requestArray = $requestParsed + $defaultRequest;
}
@ -119,14 +124,17 @@ class Request
* Constructor.
*
* @param string|array $request Query string that defines the API call (must at least contain a **method** parameter),
* eg, `'method=UserSettings.getWideScreen&idSite=1&date=yesterday&period=week&format=xml'`
* eg, `'method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week&format=xml'`
* If a request is not provided, then we use the values in the `$_GET` and `$_POST`
* superglobals.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*/
public function __construct($request = null)
public function __construct($request = null, $defaultRequest = null)
{
$this->request = self::getRequestArrayFromString($request);
$this->request = self::getRequestArrayFromString($request, $defaultRequest);
$this->sanitizeRequest();
$this->renameModuleAndActionInRequest();
}
/**
@ -134,19 +142,23 @@ class Request
* we rewrite to correct renamed plugin: Referrers
*
* @param $module
* @return string
* @param $action
* @return array( $module, $action )
* @ignore
*/
public static function renameModule($module)
public static function getRenamedModuleAndAction($module, $action)
{
$moduleToRedirect = array(
'Referers' => 'Referrers',
'PDFReports' => 'ScheduledReports',
);
if (isset($moduleToRedirect[$module])) {
return $moduleToRedirect[$module];
}
return $module;
/**
* This event is posted in the Request dispatcher and can be used
* to overwrite the Module and Action to dispatch.
* This is useful when some Controller methods or API methods have been renamed or moved to another plugin.
*
* @param $module string
* @param $action string
*/
Piwik::postEvent('Request.getRenamedModuleAndAction', array(&$module, &$action));
return array($module, $action);
}
/**
@ -168,9 +180,9 @@ class Request
/**
* Dispatches the API request to the appropriate API method and returns the result
* after post-processing.
*
*
* Post-processing includes:
*
*
* - flattening if **flat** is 0
* - running generic filters unless **disable_generic_filters** is set to 1
* - URL decoding label column values
@ -178,10 +190,10 @@ class Request
* - removing columns based on the values of the **hideColumns** and **showColumns** query parameters
* - filtering rows if the **label** query parameter is set
* - converting the result to the appropriate format (ie, XML, JSON, etc.)
*
*
* If `'original'` is supplied for the output format, the result is returned as a PHP
* object.
*
*
* @throws PluginDeactivatedException if the module plugin is not activated.
* @throws Exception if the requested API method cannot be called, if required parameters for the
* API method are missing or if the API method throws an exception and the **format**
@ -196,42 +208,90 @@ class Request
// create the response
$response = new ResponseBuilder($outputFormat, $this->request);
$corsHandler = new CORSHandler();
$corsHandler->handle();
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);
$shouldReloadAuth = false;
try {
// read parameters
$moduleMethod = Common::getRequestVar('method', null, 'string', $this->request);
list($module, $method) = $this->extractModuleAndMethod($moduleMethod);
list($module, $method) = self::getRenamedModuleAndAction($module, $method);
PluginManager::getInstance()->checkIsPluginActivated($module);
$module = $this->renameModule($module);
$apiClassName = self::getClassNameAPI($module);
if (!\Piwik\Plugin\Manager::getInstance()->isPluginActivated($module)) {
throw new PluginDeactivatedException($module);
if ($shouldReloadAuth = self::shouldReloadAuthUsingTokenAuth($this->request)) {
$access = Access::getInstance();
$tokenAuthToRestore = $access->getTokenAuth();
$hadSuperUserAccess = $access->hasSuperUserAccess();
self::forceReloadAuthUsingTokenAuth($tokenAuth);
}
$apiClassName = $this->getClassNameAPI($module);
self::reloadAuthUsingTokenAuth($this->request);
// call the method
$returnedValue = Proxy::getInstance()->call($apiClassName, $method, $this->request);
$toReturn = $response->getResponse($returnedValue, $module, $method);
} catch (Exception $e) {
Log::debug($e);
$toReturn = $response->getResponseException($e);
}
if ($shouldReloadAuth) {
$this->restoreAuthUsingTokenAuth($tokenAuthToRestore, $hadSuperUserAccess);
}
return $toReturn;
}
private function restoreAuthUsingTokenAuth($tokenToRestore, $hadSuperUserAccess)
{
// if we would not make sure to unset super user access, the tokenAuth would be not authenticated and any
// token would just keep super user access (eg if the token that was reloaded before had super user access)
Access::getInstance()->setSuperUserAccess(false);
// we need to restore by reloading the tokenAuth as some permissions could have been removed in the API
// request etc. Otherwise we could just store a clone of Access::getInstance() and restore here
self::forceReloadAuthUsingTokenAuth($tokenToRestore);
if ($hadSuperUserAccess && !Access::getInstance()->hasSuperUserAccess()) {
// we are in context of `doAsSuperUser()` and need to restore this behaviour
Access::getInstance()->setSuperUserAccess(true);
}
}
/**
* Returns the name of a plugin's API class by plugin name.
*
*
* @param string $plugin The plugin name, eg, `'Referrers'`.
* @return string The fully qualified API class name, eg, `'\Piwik\Plugins\Referrers\API'`.
*/
static public function getClassNameAPI($plugin)
public static function getClassNameAPI($plugin)
{
return sprintf('\Piwik\Plugins\%s\API', $plugin);
}
/**
* Detect if request is an API request. Meaning the module is 'API' and an API method having a valid format was
* specified.
*
* @param array $request eg array('module' => 'API', 'method' => 'Test.getMethod')
* @return bool
* @throws Exception
*/
public static function isApiRequest($request)
{
$module = Common::getRequestVar('module', '', 'string', $request);
$method = Common::getRequestVar('method', '', 'string', $request);
return $module === 'API' && !empty($method) && (count(explode('.', $method)) === 2);
}
/**
* If the token_auth is found in the $request parameter,
* the current session will be authenticated using this token_auth.
@ -241,28 +301,60 @@ class Request
* @return void
* @ignore
*/
static public function reloadAuthUsingTokenAuth($request = null)
public static function reloadAuthUsingTokenAuth($request = null)
{
// if a token_auth is specified in the API request, we load the right permissions
$token_auth = Common::getRequestVar('token_auth', '', 'string', $request);
if ($token_auth) {
/**
* Triggered when authenticating an API request, but only if the **token_auth**
* query parameter is found in the request.
*
* Plugins that provide authentication capabilities should subscribe to this event
* and make sure the global authentication object (the object returned by `Registry::get('auth')`)
* is setup to use `$token_auth` when its `authenticate()` method is executed.
*
* @param string $token_auth The value of the **token_auth** query parameter.
*/
Piwik::postEvent('API.Request.authenticate', array($token_auth));
Access::getInstance()->reloadAccess();
SettingsServer::raiseMemoryLimitIfNecessary();
if (self::shouldReloadAuthUsingTokenAuth($request)) {
self::forceReloadAuthUsingTokenAuth($token_auth);
}
}
/**
* The current session will be authenticated using this token_auth.
* It will overwrite the previous Auth object.
*
* @param string $tokenAuth
* @return void
*/
private static function forceReloadAuthUsingTokenAuth($tokenAuth)
{
/**
* Triggered when authenticating an API request, but only if the **token_auth**
* query parameter is found in the request.
*
* Plugins that provide authentication capabilities should subscribe to this event
* and make sure the global authentication object (the object returned by `StaticContainer::get('Piwik\Auth')`)
* is setup to use `$token_auth` when its `authenticate()` method is executed.
*
* @param string $token_auth The value of the **token_auth** query parameter.
*/
Piwik::postEvent('API.Request.authenticate', array($tokenAuth));
Access::getInstance()->reloadAccess();
SettingsServer::raiseMemoryLimitIfNecessary();
}
private static function shouldReloadAuthUsingTokenAuth($request)
{
if (is_null($request)) {
$request = self::getDefaultRequest();
}
if (!isset($request['token_auth'])) {
// no token is given so we just keep the current loaded user
return false;
}
// a token is specified, we need to reload auth in case it is different than the current one, even if it is empty
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $request);
// not using !== is on purpose as getTokenAuth() might return null whereas $tokenAuth is '' . In this case
// we do not need to reload.
return $tokenAuth != Access::getInstance()->getTokenAuth();
}
/**
* Returns array($class, $method) from the given string $class.$method
*
@ -286,18 +378,23 @@ class Request
* @param string $method The API method to call, ie, `'Actions.getPageTitles'`.
* @param array $paramOverride The parameter name-value pairs to use instead of what's
* in `$_GET` & `$_POST`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*
* To avoid using any parameters from $_GET or $_POST, set this to an empty `array()`.
* @return mixed The result of the API request. See {@link process()}.
*/
public static function processRequest($method, $paramOverride = array())
public static function processRequest($method, $paramOverride = array(), $defaultRequest = null)
{
$params = array();
$params['format'] = 'original';
$params['serialize'] = '0';
$params['module'] = 'API';
$params['method'] = $method;
$params = $paramOverride + $params;
// process request
$request = new Request($params);
$request = new Request($params, $defaultRequest);
return $request->process();
}
@ -305,7 +402,7 @@ class Request
* Returns the original request parameters in the current query string as an array mapping
* query parameter names with values. The result of this function will not be affected
* by any modifications to `$_GET` and will not include parameters in `$_POST`.
*
*
* @return array
*/
public static function getRequestParametersGET()
@ -343,7 +440,7 @@ class Request
// unless the filter param was in $queryParams
$genericFiltersInfo = DataTableGenericFilter::getGenericFiltersInformation();
foreach ($genericFiltersInfo as $filter) {
foreach ($filter as $queryParamName => $queryParamInfo) {
foreach ($filter[1] as $queryParamName => $queryParamInfo) {
if (!isset($params[$queryParamName])) {
$params[$queryParamName] = null;
}
@ -379,10 +476,10 @@ class Request
/**
* Returns the segment query parameter from the original request, without modifications.
*
*
* @return array|bool
*/
static public function getRawSegmentFromRequest()
public static function getRawSegmentFromRequest()
{
// we need the URL encoded segment parameter, we fetch it from _SERVER['QUERY_STRING'] instead of default URL decoded _GET
$segmentRaw = false;
@ -395,4 +492,23 @@ class Request
}
return $segmentRaw;
}
private function renameModuleAndActionInRequest()
{
if (empty($this->request['apiModule'])) {
return;
}
if (empty($this->request['apiAction'])) {
$this->request['apiAction'] = null;
}
list($this->request['apiModule'], $this->request['apiAction']) = $this->getRenamedModuleAndAction($this->request['apiModule'], $this->request['apiAction']);
}
/**
* @return array
*/
private static function getDefaultRequest()
{
return $_GET + $_POST;
}
}

506
www/analytics/core/API/ResponseBuilder.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,21 +9,22 @@
namespace Piwik\API;
use Exception;
use Piwik\API\DataTableManipulator\Flattener;
use Piwik\API\DataTableManipulator\LabelFilter;
use Piwik\API\DataTableManipulator\ReportTotalsCalculator;
use Piwik\Common;
use Piwik\DataTable\Renderer\Json;
use Piwik\DataTable\Renderer;
use Piwik\DataTable\Simple;
use Piwik\DataTable;
use Piwik\DataTable\Renderer;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\ColumnDelete;
use Piwik\DataTable\Filter\Pattern;
/**
*/
class ResponseBuilder
{
private $request = null;
private $outputFormat = null;
private $apiRenderer = null;
private $request = null;
private $sendHeader = true;
private $postProcessDataTable = true;
private $apiModule = false;
private $apiMethod = false;
@ -34,8 +35,19 @@ class ResponseBuilder
*/
public function __construct($outputFormat, $request = array())
{
$this->request = $request;
$this->outputFormat = $outputFormat;
$this->request = $request;
$this->apiRenderer = ApiRenderer::factory($outputFormat, $request);
}
public function disableSendHeader()
{
$this->sendHeader = false;
}
public function disableDataTablePostProcessor()
{
$this->postProcessDataTable = false;
}
/**
@ -70,61 +82,21 @@ class ResponseBuilder
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
if($this->outputFormat == 'original') {
@header('Content-Type: text/plain; charset=utf-8');
}
return $this->renderValue($value);
}
$this->sendHeaderIfEnabled();
/**
* Returns an error $message in the requested $format
*
* @param Exception $e
* @throws Exception
* @return string
*/
public function getResponseException(Exception $e)
{
$format = strtolower($this->outputFormat);
if ($format == 'original') {
throw $e;
}
try {
$renderer = Renderer::factory($format);
} catch (Exception $exceptionRenderer) {
return "Error: " . $e->getMessage() . " and: " . $exceptionRenderer->getMessage();
}
$e = $this->decorateExceptionWithDebugTrace($e);
$renderer->setException($e);
if ($format == 'php') {
$renderer->setSerialize($this->caseRendererPHPSerialize());
}
return $renderer->renderException();
}
/**
* @param $value
* @return string
*/
protected function renderValue($value)
{
// when null or void is returned from the api call, we handle it as a successful operation
if (!isset($value)) {
return $this->handleSuccess();
if (ob_get_contents()) {
return null;
}
return $this->apiRenderer->renderSuccess('ok');
}
// If the returned value is an object DataTable we
// apply the set of generic filters if asked in the URL
// and we render the DataTable according to the format specified in the URL
if ($value instanceof DataTable
|| $value instanceof DataTable\Map
) {
if ($value instanceof DataTableInterface) {
return $this->handleDataTable($value);
}
@ -137,26 +109,39 @@ class ResponseBuilder
return $this->handleArray($value);
}
// original data structure requested, we return without process
if ($this->outputFormat == 'original') {
return $value;
if (is_object($value)) {
return $this->apiRenderer->renderObject($value);
}
if (is_object($value)
|| is_resource($value)
) {
return $this->getResponseException(new Exception('The API cannot handle this data structure.'));
if (is_resource($value)) {
return $this->apiRenderer->renderResource($value);
}
// bool // integer // float // serialized object
return $this->handleScalar($value);
return $this->apiRenderer->renderScalar($value);
}
/**
* Returns an error $message in the requested $format
*
* @param Exception $e
* @throws Exception
* @return string
*/
public function getResponseException(Exception $e)
{
$e = $this->decorateExceptionWithDebugTrace($e);
$message = $this->formatExceptionMessage($e);
$this->sendHeaderIfEnabled();
return $this->apiRenderer->renderException($message, $e);
}
/**
* @param Exception $e
* @return Exception
*/
protected function decorateExceptionWithDebugTrace(Exception $e)
private function decorateExceptionWithDebugTrace(Exception $e)
{
// If we are in tests, show full backtrace
if (defined('PIWIK_PATH_TEST_TO_ROOT')) {
@ -165,314 +150,109 @@ class ResponseBuilder
} else {
$message = $e->getMessage() . "\n \n --> To temporarily debug this error further, set const PIWIK_PRINT_ERROR_BACKTRACE=true; in index.php";
}
return new Exception($message);
}
return $e;
}
/**
* Returns true if the user requested to serialize the output data (&serialize=1 in the request)
*
* @param mixed $defaultSerializeValue Default value in case the user hasn't specified a value
* @return bool
*/
protected function caseRendererPHPSerialize($defaultSerializeValue = 1)
private function formatExceptionMessage(Exception $exception)
{
$serialize = Common::getRequestVar('serialize', $defaultSerializeValue, 'int', $this->request);
if ($serialize) {
$message = $exception->getMessage();
if (\Piwik_ShouldPrintBackTraceWithMessage()) {
$message .= "\n" . $exception->getTraceAsString();
}
return Renderer::formatValueXml($message);
}
private function handleDataTable(DataTableInterface $datatable)
{
if ($this->postProcessDataTable) {
$postProcessor = new DataTablePostProcessor($this->apiModule, $this->apiMethod, $this->request);
$datatable = $postProcessor->process($datatable);
}
return $this->apiRenderer->renderDataTable($datatable);
}
private function handleArray($array)
{
$firstArray = null;
$firstKey = null;
if (!empty($array)) {
$firstArray = reset($array);
$firstKey = key($array);
}
$isAssoc = !empty($firstArray) && is_numeric($firstKey) && is_array($firstArray) && count(array_filter(array_keys($firstArray), 'is_string'));
if (is_numeric($firstKey)) {
$columns = Common::getRequestVar('filter_column', false, 'array', $this->request);
$pattern = Common::getRequestVar('filter_pattern', '', 'string', $this->request);
if ($columns != array(false) && $pattern !== '') {
$pattern = new Pattern(new DataTable(), $columns, $pattern);
$array = $pattern->filterArray($array);
}
$limit = Common::getRequestVar('filter_limit', -1, 'integer', $this->request);
$offset = Common::getRequestVar('filter_offset', '0', 'integer', $this->request);
if ($this->shouldApplyLimitOnArray($limit, $offset)) {
$array = array_slice($array, $offset, $limit, $preserveKeys = false);
}
}
if ($isAssoc) {
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
if ($hideColumns !== '' || $showColumns !== '') {
$columnDelete = new ColumnDelete(new DataTable(), $hideColumns, $showColumns);
$array = $columnDelete->filter($array);
}
}
return $this->apiRenderer->renderArray($array);
}
private function shouldApplyLimitOnArray($limit, $offset)
{
if ($limit === -1) {
// all fields are requested
return false;
}
if ($offset > 0) {
// an offset is specified, we have to apply the limit
return true;
}
return false;
// "api_datatable_default_limit" is set by API\Controller if no filter_limit is specified by the user.
// it holds the number of the configured default limit.
$limitSetBySystem = Common::getRequestVar('api_datatable_default_limit', -2, 'integer', $this->request);
// we ignore the limit if the datatable_default_limit was set by the system as this default filter_limit is
// only meant for dataTables but not for arrays. This way we stay BC as filter_limit was not applied pre
// Piwik 2.6 and some fixes were made in Piwik 2.13.
$wasFilterLimitSetBySystem = $limitSetBySystem !== -2;
// we check for "$limitSetBySystem === $limit" as an API method could request another API method with
// another limit. In this case we need to apply it again.
$isLimitStillDefaultLimit = $limitSetBySystem === $limit;
if ($wasFilterLimitSetBySystem && $isLimitStillDefaultLimit) {
return false;
}
return true;
}
/**
* Apply the specified renderer to the DataTable
*
* @param DataTable|array $dataTable
* @return string
*/
protected function getRenderedDataTable($dataTable)
private function sendHeaderIfEnabled()
{
$format = strtolower($this->outputFormat);
// if asked for original dataStructure
if ($format == 'original') {
// by default "original" data is not serialized
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
$dataTable = serialize($dataTable);
}
return $dataTable;
if ($this->sendHeader) {
$this->apiRenderer->sendHeader();
}
$method = Common::getRequestVar('method', '', 'string', $this->request);
$renderer = Renderer::factory($format);
$renderer->setTable($dataTable);
$renderer->setRenderSubTables(Common::getRequestVar('expanded', false, 'int', $this->request));
$renderer->setHideIdSubDatableFromResponse(Common::getRequestVar('hideIdSubDatable', false, 'int', $this->request));
if ($format == 'php') {
$renderer->setSerialize($this->caseRendererPHPSerialize());
$renderer->setPrettyDisplay(Common::getRequestVar('prettyDisplay', false, 'int', $this->request));
} else if ($format == 'html') {
$renderer->setTableId($this->request['method']);
} else if ($format == 'csv' || $format == 'tsv') {
$renderer->setConvertToUnicode(Common::getRequestVar('convertToUnicode', true, 'int', $this->request));
}
// prepare translation of column names
if ($format == 'html' || $format == 'csv' || $format == 'tsv' || $format = 'rss') {
$renderer->setApiMethod($method);
$renderer->setIdSite(Common::getRequestVar('idSite', false, 'int', $this->request));
$renderer->setTranslateColumnNames(Common::getRequestVar('translateColumnNames', false, 'int', $this->request));
}
return $renderer->render();
}
/**
* Returns a success $message in the requested $format
*
* @param string $message
* @return string
*/
protected function handleSuccess($message = 'ok')
{
// return a success message only if no content has already been buffered, useful when APIs return raw text or html content to the browser
if (!ob_get_contents()) {
switch ($this->outputFormat) {
case 'xml':
@header("Content-Type: text/xml;charset=utf-8");
$return =
"<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n" .
"<result>\n" .
"\t<success message=\"" . $message . "\" />\n" .
"</result>";
break;
case 'json':
@header("Content-Type: application/json");
$return = '{"result":"success", "message":"' . $message . '"}';
break;
case 'php':
$return = array('result' => 'success', 'message' => $message);
if ($this->caseRendererPHPSerialize()) {
$return = serialize($return);
}
break;
case 'csv':
@header("Content-Type: application/vnd.ms-excel");
@header("Content-Disposition: attachment; filename=piwik-report-export.csv");
$return = "message\n" . $message;
break;
default:
$return = 'Success:' . $message;
break;
}
return $return;
}
}
/**
* Converts the given scalar to an data table
*
* @param mixed $scalar
* @return string
*/
protected function handleScalar($scalar)
{
$dataTable = new Simple();
$dataTable->addRowsFromArray(array($scalar));
return $this->getRenderedDataTable($dataTable);
}
/**
* Handles the given data table
*
* @param DataTable $datatable
* @return string
*/
protected function handleDataTable($datatable)
{
// if requested, flatten nested tables
if (Common::getRequestVar('flat', '0', 'string', $this->request) == '1') {
$flattener = new Flattener($this->apiModule, $this->apiMethod, $this->request);
if (Common::getRequestVar('include_aggregate_rows', '0', 'string', $this->request) == '1') {
$flattener->includeAggregateRows();
}
$datatable = $flattener->flatten($datatable);
}
if (1 == Common::getRequestVar('totals', '1', 'integer', $this->request)) {
$genericFilter = new ReportTotalsCalculator($this->apiModule, $this->apiMethod, $this->request);
$datatable = $genericFilter->calculate($datatable);
}
// if the flag disable_generic_filters is defined we skip the generic filters
if (0 == Common::getRequestVar('disable_generic_filters', '0', 'string', $this->request)) {
$genericFilter = new DataTableGenericFilter($this->request);
$genericFilter->filter($datatable);
}
// we automatically safe decode all datatable labels (against xss)
$datatable->queueFilter('SafeDecodeLabel');
// if the flag disable_queued_filters is defined we skip the filters that were queued
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $this->request) == 0) {
$datatable->applyQueuedFilters();
}
// use the ColumnDelete filter if hideColumns/showColumns is provided (must be done
// after queued filters are run so processed metrics can be removed, too)
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
if ($hideColumns !== '' || $showColumns !== '') {
$datatable->filter('ColumnDelete', array($hideColumns, $showColumns));
}
// apply label filter: only return rows matching the label parameter (more than one if more than one label)
$label = $this->getLabelFromRequest($this->request);
if (!empty($label)) {
$addLabelIndex = Common::getRequestVar('labelFilterAddLabelIndex', 0, 'int', $this->request) == 1;
$filter = new LabelFilter($this->apiModule, $this->apiMethod, $this->request);
$datatable = $filter->filter($label, $datatable, $addLabelIndex);
}
return $this->getRenderedDataTable($datatable);
}
/**
* Converts the given simple array to a data table
*
* @param array $array
* @return string
*/
protected function handleArray($array)
{
if ($this->outputFormat == 'original') {
// we handle the serialization. Because some php array have a very special structure that
// couldn't be converted with the automatic DataTable->addRowsFromSimpleArray
// the user may want to request the original PHP data structure serialized by the API
// in case he has to setup serialize=1 in the URL
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
return serialize($array);
}
return $array;
}
$multiDimensional = $this->handleMultiDimensionalArray($array);
if ($multiDimensional !== false) {
return $multiDimensional;
}
return $this->getRenderedDataTable($array);
}
/**
* Is this a multi dimensional array?
* Multi dim arrays are not supported by the Datatable renderer.
* We manually render these.
*
* array(
* array(
* 1,
* 2 => array( 1,
* 2
* )
* ),
* array( 2,
* 3
* )
* );
*
* @param array $array
* @return string|bool false if it isn't a multidim array
*/
protected function handleMultiDimensionalArray($array)
{
$first = reset($array);
foreach ($array as $first) {
if (is_array($first)) {
foreach ($first as $key => $value) {
// Yes, this is a multi dim array
if (is_array($value)) {
switch ($this->outputFormat) {
case 'json':
@header("Content-Type: application/json");
return self::convertMultiDimensionalArrayToJson($array);
break;
case 'php':
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
return serialize($array);
}
return $array;
case 'xml':
@header("Content-Type: text/xml;charset=utf-8");
return $this->getRenderedDataTable($array);
default:
break;
}
}
}
}
}
return false;
}
/**
* Render a multidimensional array to Json
* Handle DataTable|Set elements in the first dimension only, following case does not work:
* array(
* array(
* DataTable,
* 2 => array(
* 1,
* 2
* ),
* ),
* );
*
* @param array $array can contain scalar, arrays, DataTable and Set
* @return string
*/
public static function convertMultiDimensionalArrayToJson($array)
{
$jsonRenderer = new Json();
$jsonRenderer->setTable($array);
$renderedReport = $jsonRenderer->render();
return $renderedReport;
}
/**
* Returns the value for the label query parameter which can be either a string
* (ie, label=...) or array (ie, label[]=...).
*
* @param array $request
* @return array
*/
static public function getLabelFromRequest($request)
{
$label = Common::getRequestVar('label', array(), 'array', $request);
if (empty($label)) {
$label = Common::getRequestVar('label', '', 'string', $request);
if (!empty($label)) {
$label = array($label);
}
}
$label = self::unsanitizeLabelParameter($label);
return $label;
}
static public function unsanitizeLabelParameter($label)
{
// this is needed because Proxy uses Common::getRequestVar which in turn
// uses Common::sanitizeInputValue. This causes the > that separates recursive labels
// to become &gt; and we need to undo that here.
$label = Common::unsanitizeInputValues($label);
return $label;
}
}

212
www/analytics/core/Access.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,55 +8,32 @@
*/
namespace Piwik;
use Piwik\Db;
use Piwik\Plugins\UsersManager\API as APIUsersManager;
use Exception;
use Piwik\Container\StaticContainer;
/**
* Singleton that manages user access to Piwik resources.
*
*
* To check whether a user has access to a resource, use one of the {@link Piwik Piwik::checkUser...}
* methods.
*
* In Piwik there are four different access levels:
*
*
* - **no access**: Users with this access level cannot view the resource.
* - **view access**: Users with this access level can view the resource, but cannot modify it.
* - **admin access**: Users with this access level can view and modify the resource.
* - **Super User access**: Only the Super User has this access level. It means the user can do
* whatever he/she wants.
*
*
* Super user access is required to set some configuration options.
* All other options are specific to the user or to a website.
*
* Access is granted per website. Uses with access for a website can view all
* data associated with that website.
*
*
*/
class Access
{
private static $instance = null;
/**
* Gets the singleton instance. Creates it if necessary.
*/
public static function getInstance()
{
if (self::$instance == null) {
self::$instance = new self;
Piwik::postEvent('Access.createAccessSingleton', array(&self::$instance));
}
return self::$instance;
}
/**
* Sets the singleton instance. For testing purposes.
*/
public static function setSingletonInstance($instance)
{
self::$instance = $instance;
}
/**
* Array of idsites available to the current user, indexed by permission level
* @see getSitesIdWith*()
@ -101,6 +78,16 @@ class Access
*/
private $auth = null;
/**
* Gets the singleton instance. Creates it if necessary.
*
* @return self
*/
public static function getInstance()
{
return StaticContainer::get('Piwik\Access');
}
/**
* Returns the list of the existing Access level.
* Useful when a given API method requests a given acccess Level.
@ -117,6 +104,11 @@ class Access
* Constructor
*/
public function __construct()
{
$this->resetSites();
}
private function resetSites()
{
$this->idsitesByAccess = array(
'view' => array(),
@ -138,15 +130,22 @@ class Access
*/
public function reloadAccess(Auth $auth = null)
{
if (!is_null($auth)) {
$this->resetSites();
if (isset($auth)) {
$this->auth = $auth;
}
// if the Auth wasn't set, we may be in the special case of setSuperUser(), otherwise we fail
if (is_null($this->auth)) {
if ($this->hasSuperUserAccess()) {
return $this->reloadAccessSuperUser();
}
if ($this->hasSuperUserAccess()) {
$this->makeSureLoginNameIsSet();
return true;
}
$this->token_auth = null;
$this->login = null;
// if the Auth wasn't set, we may be in the special case of setSuperUser(), otherwise we fail TODO: docs + review
if ($this->auth === null) {
return false;
}
@ -156,28 +155,23 @@ class Access
if (!$result->wasAuthenticationSuccessful()) {
return false;
}
$this->login = $result->getIdentity();
$this->token_auth = $result->getTokenAuth();
// case the superUser is logged in
if ($result->hasSuperUserAccess()) {
return $this->reloadAccessSuperUser();
$this->setSuperUserAccess(true);
}
// in case multiple calls to API using different tokens, we ensure we reset it as not SU
$this->setSuperUserAccess(false);
// we join with site in case there are rows in access for an idsite that doesn't exist anymore
// (backward compatibility ; before we deleted the site without deleting rows in _access table)
$accessRaw = $this->getRawSitesWithSomeViewAccess($this->login);
foreach ($accessRaw as $access) {
$this->idsitesByAccess[$access['access']][] = $access['idsite'];
}
return true;
}
public function getRawSitesWithSomeViewAccess($login)
{
return Db::fetchAll(self::getSqlAccessSite("access, t2.idsite"), $login);
$sql = self::getSqlAccessSite("access, t2.idsite");
return Db::fetchAll($sql, $login);
}
/**
@ -188,29 +182,50 @@ class Access
*/
public static function getSqlAccessSite($select)
{
return "SELECT " . $select . "
FROM " . Common::prefixTable('access') . " as t1
JOIN " . Common::prefixTable('site') . " as t2 USING (idsite) " .
" WHERE login = ?";
$access = Common::prefixTable('access');
$siteTable = Common::prefixTable('site');
return "SELECT " . $select . " FROM " . $access . " as t1
JOIN " . $siteTable . " as t2 USING (idsite) WHERE login = ?";
}
/**
* Reload Super User access
* Make sure a login name is set
*
* @return bool
* @return true
*/
protected function reloadAccessSuperUser()
protected function makeSureLoginNameIsSet()
{
$this->hasSuperUserAccess = true;
try {
$allSitesId = Plugins\SitesManager\API::getInstance()->getAllSitesId();
} catch (\Exception $e) {
$allSitesId = array();
if (empty($this->login)) {
// flag to force non empty login so Super User is not mistaken for anonymous
$this->login = 'super user was set';
}
$this->idsitesByAccess['superuser'] = $allSitesId;
}
return true;
protected function loadSitesIfNeeded()
{
if ($this->hasSuperUserAccess) {
if (empty($this->idsitesByAccess['superuser'])) {
try {
$allSitesId = Plugins\SitesManager\API::getInstance()->getAllSitesId();
} catch (\Exception $e) {
$allSitesId = array();
}
$this->idsitesByAccess['superuser'] = $allSitesId;
}
} elseif (isset($this->login)) {
if (empty($this->idsitesByAccess['view'])
&& empty($this->idsitesByAccess['admin'])) {
// we join with site in case there are rows in access for an idsite that doesn't exist anymore
// (backward compatibility ; before we deleted the site without deleting rows in _access table)
$accessRaw = $this->getRawSitesWithSomeViewAccess($this->login);
foreach ($accessRaw as $access) {
$this->idsitesByAccess[$access['access']][] = $access['idsite'];
}
}
}
}
/**
@ -221,12 +236,12 @@ class Access
*/
public function setSuperUserAccess($bool = true)
{
if ($bool) {
$this->reloadAccessSuperUser();
} else {
$this->hasSuperUserAccess = false;
$this->idsitesByAccess['superuser'] = array();
$this->hasSuperUserAccess = (bool) $bool;
if ($bool) {
$this->makeSureLoginNameIsSet();
} else {
$this->resetSites();
}
}
@ -269,6 +284,8 @@ class Access
*/
public function getSitesIdWithAtLeastViewAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['view'],
$this->idsitesByAccess['admin'],
@ -284,13 +301,14 @@ class Access
*/
public function getSitesIdWithAdminAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['admin'],
$this->idsitesByAccess['superuser'])
);
}
/**
* Returns an array of ID sites for which the user has a VIEW access only.
*
@ -300,6 +318,8 @@ class Access
*/
public function getSitesIdWithViewAccess()
{
$this->loadSitesIfNeeded();
return $this->idsitesByAccess['view'];
}
@ -315,6 +335,22 @@ class Access
}
}
/**
* Returns `true` if the current user has admin access to at least one site.
*
* @return bool
*/
public function isUserHasSomeAdminAccess()
{
if ($this->hasSuperUserAccess()) {
return true;
}
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
return count($idSitesAccessible) > 0;
}
/**
* If the user doesn't have an ADMIN access for at least one website, throws an exception
*
@ -322,11 +358,7 @@ class Access
*/
public function checkUserHasSomeAdminAccess()
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
if (count($idSitesAccessible) == 0) {
if (!$this->isUserHasSomeAdminAccess()) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('admin')));
}
}
@ -341,7 +373,9 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
if (count($idSitesAccessible) == 0) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('view')));
}
@ -359,8 +393,10 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'admin'", $idsite)));
@ -380,8 +416,10 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'view'", $idsite)));
@ -401,11 +439,41 @@ class Access
}
$idSites = Site::getIdSitesFromIdSitesString($idSites);
if (empty($idSites)) {
throw new NoAccessException("The parameter 'idSite=' is missing from the request.");
}
return $idSites;
}
/**
* Executes a callback with superuser privileges, making sure those privileges are rescinded
* before this method exits. Privileges will be rescinded even if an exception is thrown.
*
* @param callback $function The callback to execute. Should accept no arguments.
* @return mixed The result of `$function`.
* @throws Exception rethrows any exceptions thrown by `$function`.
* @api
*/
public static function doAsSuperUser($function)
{
$isSuperUser = self::getInstance()->hasSuperUserAccess();
self::getInstance()->setSuperUserAccess(true);
try {
$result = $function();
} catch (Exception $ex) {
self::getInstance()->setSuperUserAccess($isSuperUser);
throw $ex;
}
self::getInstance()->setSuperUserAccess($isSuperUser);
return $result;
}
}
/**

246
www/analytics/core/Application/Environment.php Normal file
View file

@ -0,0 +1,246 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use DI\Container;
use Piwik\Application\Kernel\EnvironmentValidator;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
use Piwik\Container\ContainerFactory;
use Piwik\Container\StaticContainer;
use Piwik\Piwik;
/**
* Encapsulates Piwik environment setup and access.
*
* The Piwik environment consists of two main parts: the kernel and the DI container.
*
* The 'kernel' is the core part of Piwik that cannot be modified / extended through the DI container.
* It includes components that are required to create the DI container.
*
* Currently the only objects in the 'kernel' are a GlobalSettingsProvider object and a
* PluginList object. The GlobalSettingsProvider object is required for the current PluginList
* implementation and for checking whether Development mode is enabled. The PluginList is
* needed in order to determine what plugins are activated, since plugins can provide their
* own DI configuration.
*
* The DI container contains every other Piwik object, including the Plugin\Manager,
* plugin API instances, dependent services, etc. Plugins and users can override/extend
* the objects in this container.
*
* NOTE: DI support in Piwik is currently a work in process; not everything is currently
* stored in the DI container, but we are working towards this.
*/
class Environment
{
/**
* @internal
* @var EnvironmentManipulator
*/
private static $globalEnvironmentManipulator = null;
/**
* @var string
*/
private $environment;
/**
* @var array
*/
private $definitions;
/**
* @var Container
*/
private $container;
/**
* @var GlobalSettingsProvider
*/
private $globalSettingsProvider;
/**
* @var PluginList
*/
private $pluginList;
/**
* @param string $environment
* @param array $definitions
*/
public function __construct($environment, array $definitions = array())
{
$this->environment = $environment;
$this->definitions = $definitions;
}
/**
* Initializes the kernel globals and DI container.
*/
public function init()
{
$this->invokeBeforeContainerCreatedHook();
$this->container = $this->createContainer();
StaticContainer::push($this->container);
$this->validateEnvironment();
$this->invokeEnvironmentBootstrappedHook();
Piwik::postEvent('Environment.bootstrapped'); // this event should be removed eventually
}
/**
* Destroys an environment. MUST be called when embedding environments.
*/
public function destroy()
{
StaticContainer::pop();
}
/**
* Returns the DI container. All Piwik objects for a specific Piwik instance should be stored
* in this container.
*
* @return Container
*/
public function getContainer()
{
return $this->container;
}
/**
* @link http://php-di.org/doc/container-configuration.html
*/
private function createContainer()
{
$pluginList = $this->getPluginListCached();
$settings = $this->getGlobalSettingsCached();
$extraDefinitions = $this->getExtraDefinitionsFromManipulators();
$definitions = array_merge(StaticContainer::getDefinitions(), $extraDefinitions, array($this->definitions));
$environments = array($this->environment);
$environments = array_merge($environments, $this->getExtraEnvironmentsFromManipulators());
$containerFactory = new ContainerFactory($pluginList, $settings, $environments, $definitions);
return $containerFactory->create();
}
protected function getGlobalSettingsCached()
{
if ($this->globalSettingsProvider === null) {
$original = $this->getGlobalSettings();
$globalSettingsProvider = $this->getGlobalSettingsProviderOverride($original);
$this->globalSettingsProvider = $globalSettingsProvider ?: $original;
}
return $this->globalSettingsProvider;
}
protected function getPluginListCached()
{
if ($this->pluginList === null) {
$pluginList = $this->getPluginListOverride();
$this->pluginList = $pluginList ?: $this->getPluginList();
}
return $this->pluginList;
}
/**
* Returns the kernel global GlobalSettingsProvider object. Derived classes can override this method
* to provide a different implementation.
*
* @return null|GlobalSettingsProvider
*/
protected function getGlobalSettings()
{
return new GlobalSettingsProvider();
}
/**
* Returns the kernel global PluginList object. Derived classes can override this method to
* provide a different implementation.
*
* @return PluginList
*/
protected function getPluginList()
{
// TODO: in tracker should only load tracker plugins. can't do properly until tracker entrypoint is encapsulated.
return new PluginList($this->getGlobalSettingsCached());
}
private function validateEnvironment()
{
/** @var EnvironmentValidator $validator */
$validator = $this->container->get('Piwik\Application\Kernel\EnvironmentValidator');
$validator->validate();
}
/**
* @param EnvironmentManipulator $manipulator
* @internal
*/
public static function setGlobalEnvironmentManipulator(EnvironmentManipulator $manipulator)
{
self::$globalEnvironmentManipulator = $manipulator;
}
private function getGlobalSettingsProviderOverride(GlobalSettingsProvider $original)
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makeGlobalSettingsProvider($original);
} else {
return null;
}
}
private function invokeBeforeContainerCreatedHook()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->beforeContainerCreated();
}
}
private function getExtraDefinitionsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraDefinitions();
} else {
return array();
}
}
private function invokeEnvironmentBootstrappedHook()
{
if (self::$globalEnvironmentManipulator) {
self::$globalEnvironmentManipulator->onEnvironmentBootstrapped();
}
}
private function getExtraEnvironmentsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraEnvironments();
} else {
return array();
}
}
private function getPluginListOverride()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makePluginList($this->getGlobalSettingsCached());
} else {
return null;
}
}
}

59
www/analytics/core/Application/EnvironmentManipulator.php Normal file
View file

@ -0,0 +1,59 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
/**
* Used to manipulate Environment instances before the container is created.
* Only used by the testing environment setup code, shouldn't be used anywhere
* else.
*/
interface EnvironmentManipulator
{
/**
* Create a custom GlobalSettingsProvider kernel object, overriding the default behavior.
*
* @return GlobalSettingsProvider
*/
public function makeGlobalSettingsProvider(GlobalSettingsProvider $original);
/**
* Create a custom PluginList kernel object, overriding the default behavior.@deprecated
*
* @param GlobalSettingsProvider $globalSettingsProvider
* @return PluginList
*/
public function makePluginList(GlobalSettingsProvider $globalSettingsProvider);
/**
* Invoked before the container is created.
*/
public function beforeContainerCreated();
/**
* Return an array of definition arrays that override DI config specified in PHP config files.
*
* @return array[]
*/
public function getExtraDefinitions();
/**
* Invoked after the container is created and the environment is considered bootstrapped.
*/
public function onEnvironmentBootstrapped();
/**
* Return an array of environment names to apply after the normal environment.
*
* @return string[]
*/
public function getExtraEnvironments();
}

79
www/analytics/core/Application/Kernel/EnvironmentValidator.php Normal file
View file

@ -0,0 +1,79 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Common;
use Piwik\Piwik;
use Piwik\SettingsServer;
use Piwik\Translation\Translator;
/**
* Validates the Piwik environment. This includes making sure the required config files
* are present, and triggering the correct behaviour if otherwise.
*/
class EnvironmentValidator
{
/**
* @var GlobalSettingsProvider
*/
protected $settingsProvider;
/**
* @var Translator
*/
protected $translator;
public function __construct(GlobalSettingsProvider $settingsProvider, Translator $translator)
{
$this->settingsProvider = $settingsProvider;
$this->translator = $translator;
}
public function validate()
{
$inTrackerRequest = SettingsServer::isTrackerApiRequest();
$inConsole = Common::isPhpCliMode();
$this->checkConfigFileExists($this->settingsProvider->getPathGlobal());
$this->checkConfigFileExists($this->settingsProvider->getPathLocal(), $startInstaller = !$inTrackerRequest && !$inConsole);
}
/**
* @param $path
* @param bool $startInstaller
* @throws \Exception
*/
private function checkConfigFileExists($path, $startInstaller = false)
{
if (is_readable($path)) {
return;
}
$message = $this->translator->translate('General_ExceptionConfigurationFileNotFound', array($path));
if (Common::isPhpCliMode()) {
$message .= "\n" . $this->translator->translate('General_ExceptionConfigurationFileNotFound2', array($path, get_current_user()));
}
$exception = new \Exception($message);
if ($startInstaller) {
/**
* Triggered when the configuration file cannot be found or read, which usually
* means Piwik is not installed yet.
*
* This event can be used to start the installation process or to display a custom error message.
*
* @param \Exception $exception The exception that was thrown by `Config::getInstance()`.
*/
Piwik::postEvent('Config.NoConfigurationFile', array($exception), $pending = true);
} else {
throw $exception;
}
}
}

111
www/analytics/core/Application/Kernel/GlobalSettingsProvider.php Normal file
View file

@ -0,0 +1,111 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Config;
use Piwik\Config\IniFileChain;
/**
* Provides global settings. Global settings are organized in sections where
* each section contains a list of name => value pairs. Setting values can
* be primitive values or arrays of primitive values.
*
* Uses the config.ini.php, common.ini.php and global.ini.php files to provide global settings.
*
* At the moment a singleton instance of this class is used in order to get tests to pass.
*/
class GlobalSettingsProvider
{
/**
* @var IniFileChain
*/
protected $iniFileChain;
/**
* @var string
*/
protected $pathGlobal = null;
/**
* @var string
*/
protected $pathCommon = null;
/**
* @var string
*/
protected $pathLocal = null;
/**
* @param string|null $pathGlobal Path to the global.ini.php file. Or null to use the default.
* @param string|null $pathLocal Path to the config.ini.php file. Or null to use the default.
* @param string|null $pathCommon Path to the common.ini.php file. Or null to use the default.
*/
public function __construct($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: Config::getGlobalConfigPath();
$this->pathCommon = $pathCommon ?: Config::getCommonConfigPath();
$this->pathLocal = $pathLocal ?: Config::getLocalConfigPath();
$this->iniFileChain = new IniFileChain();
$this->reload();
}
public function reload($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: $this->pathGlobal;
$this->pathCommon = $pathCommon ?: $this->pathCommon;
$this->pathLocal = $pathLocal ?: $this->pathLocal;
$this->iniFileChain->reload(array($this->pathGlobal, $this->pathCommon), $this->pathLocal);
}
/**
* Returns a settings section.
*
* @param string $name
* @return array
*/
public function &getSection($name)
{
$section =& $this->iniFileChain->get($name);
return $section;
}
/**
* Sets a settings section.
*
* @param string $name
* @param array $value
*/
public function setSection($name, $value)
{
$this->iniFileChain->set($name, $value);
}
public function getIniFileChain()
{
return $this->iniFileChain;
}
public function getPathGlobal()
{
return $this->pathGlobal;
}
public function getPathLocal()
{
return $this->pathLocal;
}
public function getPathCommon()
{
return $this->pathCommon;
}
}

120
www/analytics/core/Application/Kernel/PluginList.php Normal file
View file

@ -0,0 +1,120 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
/**
* Lists the currently activated plugins. Used when setting up Piwik's environment before
* initializing the DI container.
*
* Uses the [Plugins] section in Piwik's INI config to get the activated plugins.
*
* Depends on GlobalSettingsProvider being used.
*
* TODO: parts of Plugin\Manager edit the plugin list; maybe PluginList implementations should be mutable?
*/
class PluginList
{
/**
* @var GlobalSettingsProvider
*/
private $settings;
/**
* Plugins bundled with core package, disabled by default
* @var array
*/
private $corePluginsDisabledByDefault = array(
'DBStats',
'ExampleCommand',
'ExampleSettingsPlugin',
'ExampleUI',
'ExampleVisualization',
'ExamplePluginTemplate',
'ExampleTracker',
'ExampleReport',
'MobileAppMeasurable',
'Provider'
);
// Themes bundled with core package, disabled by default
private $coreThemesDisabledByDefault = array(
'ExampleTheme'
);
public function __construct(GlobalSettingsProvider $settings)
{
$this->settings = $settings;
}
/**
* Returns the list of plugins that should be loaded. Used by the container factory to
* load plugin specific DI overrides.
*
* @return string[]
*/
public function getActivatedPlugins()
{
$section = $this->settings->getSection('Plugins');
return @$section['Plugins'] ?: array();
}
/**
* Returns the list of plugins that are bundled with Piwik.
*
* @return string[]
*/
public function getPluginsBundledWithPiwik()
{
$pathGlobal = $this->settings->getPathGlobal();
$section = $this->settings->getIniFileChain()->getFrom($pathGlobal, 'Plugins');
return $section['Plugins'];
}
/**
* Returns the plugins bundled with core package that are disabled by default.
*
* @return string[]
*/
public function getCorePluginsDisabledByDefault()
{
return array_merge($this->corePluginsDisabledByDefault, $this->coreThemesDisabledByDefault);
}
/**
* Sorts an array of plugins in the order they should be loaded.
*
* @params string[] $plugins
* @return \string[]
*/
public function sortPlugins(array $plugins)
{
$global = $this->getPluginsBundledWithPiwik();
if (empty($global)) {
return $plugins;
}
// we need to make sure a possibly disabled plugin will be still loaded before any 3rd party plugin
$global = array_merge($global, $this->corePluginsDisabledByDefault);
$global = array_values($global);
$plugins = array_values($plugins);
$defaultPluginsLoadedFirst = array_intersect($global, $plugins);
$otherPluginsToLoadAfterDefaultPlugins = array_diff($plugins, $defaultPluginsLoadedFirst);
// sort by name to have a predictable order for those extra plugins
sort($otherPluginsToLoadAfterDefaultPlugins);
$sorted = array_merge($defaultPluginsLoadedFirst, $otherPluginsToLoadAfterDefaultPlugins);
return $sorted;
}
}

421
www/analytics/core/Archive.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,57 +9,58 @@
namespace Piwik;
use Piwik\Archive\Parameters;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Archive\ArchiveInvalidator;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\Period\Range;
use Piwik\Period\Factory as PeriodFactory;
/**
* The **Archive** class is used to query cached analytics statistics
* (termed "archive data").
*
*
* You can use **Archive** instances to get data that was archived for one or more sites,
* for one or more periods and one optional segment.
*
*
* If archive data is not found, this class will initiate the archiving process. [1](#footnote-1)
*
*
* **Archive** instances must be created using the {@link build()} factory method;
* they cannot be constructed.
*
*
* You can search for metrics (such as `nb_visits`) using the {@link getNumeric()} and
* {@link getDataTableFromNumeric()} methods. You can search for
* reports using the {@link getBlob()}, {@link getDataTable()} and {@link getDataTableExpanded()} methods.
*
*
* If you're creating an API that returns report data, you may want to use the
* {@link getDataTableFromArchive()} helper function.
*
*
* ### Learn more
*
*
* Learn more about _archiving_ [here](/guides/all-about-analytics-data).
*
*
* ### Limitations
*
*
* - You cannot get data for multiple range periods in a single query.
* - You cannot get data for periods of different types in a single query.
*
*
* ### Examples
*
*
* **_Querying metrics for an API method_**
*
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* // all sites and multiple dates
* $archive = Archive::build($idSite = 'all', $period = 'month', $date = '2013-01-02,2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* **_Querying and using metrics immediately_**
*
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $data = $archive->getNumeric(array('nb_visits', 'nb_actions'));
*
*
* $visits = $data['nb_visits'];
* $actions = $data['nb_actions'];
*
@ -68,41 +69,40 @@ use Piwik\Period\Range;
* // multiple sites and multiple dates
* $archive = Archive::build($idSite = '1,2,3', $period = 'month', $date = '2013-01-02,2013-03-08');
* $data = $archive->getNumeric('nb_visits');
*
*
* $janSite1Visits = $data['1']['2013-01-01,2013-01-31']['nb_visits'];
* $febSite1Visits = $data['1']['2013-02-01,2013-02-28']['nb_visits'];
* // ... etc.
*
*
* **_Querying for reports_**
*
*
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $dataTable = $archive->getDataTable('MyPlugin_MyReport');
* // ... manipulate $dataTable ...
* return $dataTable;
*
*
* **_Querying a report for an API method_**
*
*
* public function getMyReport($idSite, $period, $date, $segment = false, $expanded = false)
* {
* $dataTable = Archive::getDataTableFromArchive('MyPlugin_MyReport', $idSite, $period, $date, $segment, $expanded);
* $dataTable->queueFilter('ReplaceColumnNames');
* return $dataTable;
* }
*
*
* **_Querying data for multiple range periods_**
*
*
* // get data for first range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-08,2013-03-12');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* // get data for second range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-15,2013-03-20');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* <a name="footnote-1"></a>
* [1]: The archiving process will not be launched if browser archiving is disabled
* and the current request came from a browser (and not the **archive.php** cron
* script).
* and the current request came from a browser.
*
*
* @api
@ -162,6 +162,16 @@ class Archive
*/
private $params;
/**
* @var \Piwik\Cache\Cache
*/
private static $cache;
/**
* @var ArchiveInvalidator
*/
private $invalidator;
/**
* @param Parameters $params
* @param bool $forceIndexedBySite Whether to force index the result of a query by site ID.
@ -173,6 +183,8 @@ class Archive
$this->params = $params;
$this->forceIndexedBySite = $forceIndexedBySite;
$this->forceIndexedByDate = $forceIndexedByDate;
$this->invalidator = StaticContainer::get('Piwik\Archive\ArchiveInvalidator');
}
/**
@ -192,37 +204,42 @@ class Archive
* or date range (ie, 'YYYY-MM-DD,YYYY-MM-DD').
* @param bool|false|string $segment Segment definition or false if no segment should be used. {@link Piwik\Segment}
* @param bool|false|string $_restrictSitesToLogin Used only when running as a scheduled task.
* @param bool $skipAggregationOfSubTables Whether the archive, when it is processed, should also aggregate all sub-tables
* @return Archive
* @return static
*/
public static function build($idSites, $period, $strDate, $segment = false, $_restrictSitesToLogin = false, $skipAggregationOfSubTables = false)
public static function build($idSites, $period, $strDate, $segment = false, $_restrictSitesToLogin = false)
{
$websiteIds = Site::getIdSitesFromIdSitesString($idSites, $_restrictSitesToLogin);
$timezone = false;
if (count($websiteIds) == 1) {
$timezone = Site::getTimezoneFor($websiteIds[0]);
}
if (Period::isMultiplePeriod($strDate, $period)) {
$oPeriod = new Range($period, $strDate);
$oPeriod = PeriodFactory::build($period, $strDate, $timezone);
$allPeriods = $oPeriod->getSubperiods();
} else {
$timezone = count($websiteIds) == 1 ? Site::getTimezoneFor($websiteIds[0]) : false;
$oPeriod = Period::makePeriodFromQueryParams($timezone, $period, $strDate);
$oPeriod = PeriodFactory::makePeriodFromQueryParams($timezone, $period, $strDate);
$allPeriods = array($oPeriod);
}
$segment = new Segment($segment, $websiteIds);
$idSiteIsAll = $idSites == self::REQUEST_ALL_WEBSITES_FLAG;
$segment = new Segment($segment, $websiteIds);
$idSiteIsAll = $idSites == self::REQUEST_ALL_WEBSITES_FLAG;
$isMultipleDate = Period::isMultiplePeriod($strDate, $period);
return Archive::factory($segment, $allPeriods, $websiteIds, $idSiteIsAll, $isMultipleDate, $skipAggregationOfSubTables);
return static::factory($segment, $allPeriods, $websiteIds, $idSiteIsAll, $isMultipleDate);
}
/**
* Returns a new Archive instance that will query archive data for the given set of
* sites and periods, using an optional segment.
*
*
* This method uses an array of Period instances and a Segment instance, instead of strings
* like {@link build()}.
*
*
* If you want to create an Archive instance using data found in query parameters,
* use {@link build()}.
*
*
* @param Segment $segment The segment to use. For no segment, use `new Segment('', $idSites)`.
* @param array $periods An array of Period instances.
* @param array $idSites An array of site IDs (eg, `array(1, 2, 3)`).
@ -232,41 +249,42 @@ class Archive
* @param bool $isMultipleDate Whether multiple dates are being queried or not. If true, then
* the result of querying functions will be indexed by period,
* regardless of whether `count($periods) == 1`.
* @param bool $skipAggregationOfSubTables Whether the archive should skip aggregation of all sub-tables
*
* @return Archive
*/
public static function factory(Segment $segment, array $periods, array $idSites, $idSiteIsAll = false, $isMultipleDate = false, $skipAggregationOfSubTables = false)
public static function factory(Segment $segment, array $periods, array $idSites, $idSiteIsAll = false, $isMultipleDate = false)
{
$forceIndexedBySite = false;
$forceIndexedByDate = false;
if ($idSiteIsAll || count($idSites) > 1) {
$forceIndexedBySite = true;
}
if (count($periods) > 1 || $isMultipleDate) {
$forceIndexedByDate = true;
}
$params = new Parameters($idSites, $periods, $segment, $skipAggregationOfSubTables);
$params = new Parameters($idSites, $periods, $segment);
return new Archive($params, $forceIndexedBySite, $forceIndexedByDate);
return new static($params, $forceIndexedBySite, $forceIndexedByDate);
}
/**
* Queries and returns metric data in an array.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be indexed by site ID first, then period.
*
*
* @param string|array $names One or more archive names, eg, `'nb_visits'`, `'Referrers_distinctKeywords'`,
* etc.
* @return false|numeric|array `false` if there is no data to return, a single numeric value if we're not querying
* @return false|integer|array `false` if there is no data to return, a single numeric value if we're not querying
* for multiple sites/periods, or an array if multiple sites, periods or names are
* queried for.
*/
@ -288,50 +306,22 @@ class Archive
return $result;
}
/**
* Queries and returns blob data in an array.
*
* Reports are stored in blobs as serialized arrays of {@link DataTable\Row} instances, but this
* data can technically be anything. In other words, you can store whatever you want
* as archive data blobs.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be indexed by site ID first, then period.
*
* @param string|array $names One or more archive names, eg, `'Referrers_keywordBySearchEngine'`.
* @param null|string $idSubtable If we're returning serialized DataTable data, then this refers
* to the subtable ID to return. If set to 'all', all subtables
* of each requested report are returned.
* @return array An array of appropriately indexed blob data.
*/
public function getBlob($names, $idSubtable = null)
{
$data = $this->get($names, 'blob', $idSubtable);
return $data->getIndexedArray($this->getResultIndices());
}
/**
* Queries and returns metric data in a DataTable instance.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
*
* _Note: Every DataTable instance returned will have at most one row in it. The contents of each
* row will be the requested metrics for the appropriate site and period._
*
*
* @param string|array $names One or more archive names, eg, 'nb_visits', 'Referrers_distinctKeywords',
* etc.
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
@ -343,22 +333,40 @@ class Archive
return $data->getDataTable($this->getResultIndices());
}
/**
* Similar to {@link getDataTableFromNumeric()} but merges all children on the created DataTable.
*
* This is the same as doing `$this->getDataTableFromNumeric()->mergeChildren()` but this way it is much faster.
*
* @return DataTable|DataTable\Map
*
* @internal Currently only used by MultiSites.getAll plugin. Feel free to remove internal tag if needed somewhere
* else. If no longer needed by MultiSites.getAll please remove this method. If you need this to work in
* a bit different way feel free to refactor as always.
*/
public function getDataTableFromNumericAndMergeChildren($names)
{
$data = $this->get($names, 'numeric');
$resultIndexes = $this->getResultIndices();
return $data->getMergedDataTable($resultIndexes);
}
/**
* Queries and returns one or more reports as DataTable instances.
*
*
* This method will query blob data that is a serialized array of of {@link DataTable\Row}'s and
* unserialize it.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
*
* @param string $name The name of the record to get. This method can only query one record at a time.
* @param int|string|null $idSubtable The ID of the subtable to get (if any).
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
@ -372,13 +380,13 @@ class Archive
/**
* Queries and returns one report with all of its subtables loaded.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map indexed} by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
@ -400,16 +408,18 @@ class Archive
/**
* Returns the list of plugins that archive the given reports.
*
*
* @param array $archiveNames
* @return array
*/
private function getRequestedPlugins($archiveNames)
{
$result = array();
foreach ($archiveNames as $name) {
$result[] = self::getPluginForReport($name);
}
return array_unique($result);
}
@ -427,7 +437,7 @@ class Archive
/**
* Helper function that creates an Archive instance and queries for report data using
* query parameter data. API methods can use this method to reduce code redundancy.
*
*
* @param string $name The name of the report to return.
* @param int|string|array $idSite @see {@link build()}
* @param string $period @see {@link build()}
@ -435,21 +445,19 @@ class Archive
* @param string $segment @see {@link build()}
* @param bool $expanded If true, loads all subtables. See {@link getDataTableExpanded()}
* @param int|null $idSubtable See {@link getDataTableExpanded()}
* @param bool $skipAggregationOfSubTables Whether or not we should skip the aggregation of all sub-tables and only aggregate parent DataTable.
* @param int|null $depth See {@link getDataTableExpanded()}
* @throws \Exception
* @return DataTable|DataTable\Map See {@link getDataTable()} and
* {@link getDataTableExpanded()} for more
* information
* @deprecated Since Piwik 2.12.0 Use Archive::createDataTableFromArchive() instead
*/
public static function getDataTableFromArchive($name, $idSite, $period, $date, $segment, $expanded,
$idSubtable = null, $skipAggregationOfSubTables = false, $depth = null)
$idSubtable = null, $depth = null)
{
Piwik::checkUserHasViewAccess($idSite);
if($skipAggregationOfSubTables && ($expanded || $idSubtable)) {
throw new \Exception("Not expected to skipAggregationOfSubTables when expanded=1 or idSubtable is set.");
}
$archive = Archive::build($idSite, $period, $date, $segment, $_restrictSitesToLogin = false, $skipAggregationOfSubTables);
$archive = Archive::build($idSite, $period, $date, $segment, $_restrictSitesToLogin = false);
if ($idSubtable === false) {
$idSubtable = null;
}
@ -465,9 +473,102 @@ class Archive
return $dataTable;
}
private function appendIdSubtable($recordName, $id)
/**
* Helper function that creates an Archive instance and queries for report data using
* query parameter data. API methods can use this method to reduce code redundancy.
*
* @param string $recordName The name of the report to return.
* @param int|string|array $idSite @see {@link build()}
* @param string $period @see {@link build()}
* @param string $date @see {@link build()}
* @param string $segment @see {@link build()}
* @param bool $expanded If true, loads all subtables. See {@link getDataTableExpanded()}
* @param bool $flat If true, loads all subtables and disabled all recursive filters.
* @param int|null $idSubtable See {@link getDataTableExpanded()}
* @param int|null $depth See {@link getDataTableExpanded()}
* @return DataTable|DataTable\Map
*/
public static function createDataTableFromArchive($recordName, $idSite, $period, $date, $segment, $expanded = false, $flat = false, $idSubtable = null, $depth = null)
{
return $recordName . "_" . $id;
if ($flat && !$idSubtable) {
$expanded = true;
}
$dataTable = self::getDataTableFromArchive($recordName, $idSite, $period, $date, $segment, $expanded, $idSubtable, $depth);
$dataTable->queueFilter('ReplaceColumnNames');
if ($expanded) {
$dataTable->queueFilterSubtables('ReplaceColumnNames');
}
if ($flat) {
$dataTable->disableRecursiveFilters();
}
return $dataTable;
}
private function getSiteIdsThatAreRequestedInThisArchiveButWereNotInvalidatedYet()
{
if (is_null(self::$cache)) {
self::$cache = Cache::getTransientCache();
}
$id = 'Archive.SiteIdsOfRememberedReportsInvalidated';
if (!self::$cache->contains($id)) {
self::$cache->save($id, array());
}
$siteIdsAlreadyHandled = self::$cache->fetch($id);
$siteIdsRequested = $this->params->getIdSites();
foreach ($siteIdsRequested as $index => $siteIdRequested) {
$siteIdRequested = (int) $siteIdRequested;
if (in_array($siteIdRequested, $siteIdsAlreadyHandled)) {
unset($siteIdsRequested[$index]); // was already handled previously, do not do it again
} else {
$siteIdsAlreadyHandled[] = $siteIdRequested; // we will handle this id this time
}
}
self::$cache->save($id, $siteIdsAlreadyHandled);
return $siteIdsRequested;
}
private function invalidatedReportsIfNeeded()
{
$siteIdsRequested = $this->getSiteIdsThatAreRequestedInThisArchiveButWereNotInvalidatedYet();
if (empty($siteIdsRequested)) {
return; // all requested site ids were already handled
}
$sitesPerDays = $this->invalidator->getRememberedArchivedReportsThatShouldBeInvalidated();
foreach ($sitesPerDays as $date => $siteIds) {
if (empty($siteIds)) {
continue;
}
$siteIdsToActuallyInvalidate = array_intersect($siteIds, $siteIdsRequested);
if (empty($siteIdsToActuallyInvalidate)) {
continue; // all site ids that should be handled are already handled
}
try {
$this->invalidator->markArchivesAsInvalidated($siteIdsToActuallyInvalidate, array(Date::factory($date)), false);
} catch (\Exception $e) {
Site::clearCache();
throw $e;
}
}
Site::clearCache();
}
/**
@ -477,7 +578,7 @@ class Archive
* @param null|int $idSubtable
* @return Archive\DataCollection
*/
private function get($archiveNames, $archiveDataType, $idSubtable = null)
protected function get($archiveNames, $archiveDataType, $idSubtable = null)
{
if (!is_array($archiveNames)) {
$archiveNames = array($archiveNames);
@ -487,35 +588,41 @@ class Archive
if ($idSubtable !== null
&& $idSubtable != self::ID_SUBTABLE_LOAD_ALL_SUBTABLES
) {
foreach ($archiveNames as &$name) {
$name = $this->appendIdsubtable($name, $idSubtable);
// this is also done in ArchiveSelector. It should be actually only done in ArchiveSelector but DataCollection
// does require to have the subtableId appended. Needs to be changed in refactoring to have it only in one
// place.
$dataNames = array();
foreach ($archiveNames as $name) {
$dataNames[] = ArchiveSelector::appendIdsubtable($name, $idSubtable);
}
} else {
$dataNames = $archiveNames;
}
$result = new Archive\DataCollection(
$archiveNames, $archiveDataType, $this->params->getIdSites(), $this->params->getPeriods(), $defaultRow = null);
$dataNames, $archiveDataType, $this->params->getIdSites(), $this->params->getPeriods(), $defaultRow = null);
$archiveIds = $this->getArchiveIds($archiveNames);
if (empty($archiveIds)) {
return $result;
}
$loadAllSubtables = $idSubtable == self::ID_SUBTABLE_LOAD_ALL_SUBTABLES;
$archiveData = ArchiveSelector::getArchiveData($archiveIds, $archiveNames, $archiveDataType, $loadAllSubtables);
$archiveData = ArchiveSelector::getArchiveData($archiveIds, $archiveNames, $archiveDataType, $idSubtable);
$isNumeric = $archiveDataType == 'numeric';
foreach ($archiveData as $row) {
// values are grouped by idsite (site ID), date1-date2 (date range), then name (field name)
$idSite = $row['idsite'];
$periodStr = $row['date1'] . "," . $row['date2'];
$periodStr = $row['date1'] . ',' . $row['date2'];
if ($archiveDataType == 'numeric') {
$value = $this->formatNumericValue($row['value']);
if ($isNumeric) {
$row['value'] = $this->formatNumericValue($row['value']);
} else {
$value = $this->uncompress($row['value']);
$result->addMetadata($idSite, $periodStr, 'ts_archived', $row['ts_archived']);
$result->addMetadata($row['idsite'], $periodStr, DataTable::ARCHIVED_DATE_METADATA_NAME, $row['ts_archived']);
}
$resultRow = & $result->get($idSite, $periodStr);
$resultRow[$row['name']] = $value;
$result->set($row['idsite'], $periodStr, $row['name'], $row['value']);
}
return $result;
@ -526,6 +633,9 @@ class Archive
* queried. This function will use the idarchive cache if it has the right data,
* query archive tables for IDs w/o launching archiving, or launch archiving and
* get the idarchive from ArchiveProcessor instances.
*
* @param string $archiveNames
* @return array
*/
private function getArchiveIds($archiveNames)
{
@ -533,7 +643,7 @@ class Archive
// figure out which archives haven't been processed (if an archive has been processed,
// then we have the archive IDs in $this->idarchives)
$doneFlags = array();
$doneFlags = array();
$archiveGroups = array();
foreach ($plugins as $plugin) {
$doneFlag = $this->getDoneStringForPlugin($plugin);
@ -542,7 +652,7 @@ class Archive
if (!isset($this->idarchives[$doneFlag])) {
$archiveGroup = $this->getArchiveGroupOfPlugin($plugin);
if($archiveGroup == self::ARCHIVE_ALL_PLUGINS_FLAG) {
if ($archiveGroup == self::ARCHIVE_ALL_PLUGINS_FLAG) {
$archiveGroup = reset($plugins);
}
$archiveGroups[] = $archiveGroup;
@ -560,19 +670,7 @@ class Archive
}
}
// order idarchives by the table month they belong to
$idArchivesByMonth = array();
foreach (array_keys($doneFlags) as $doneFlag) {
if (empty($this->idarchives[$doneFlag])) {
continue;
}
foreach ($this->idarchives[$doneFlag] as $dateRange => $idarchives) {
foreach ($idarchives as $id) {
$idArchivesByMonth[$dateRange][] = $id;
}
}
}
$idArchivesByMonth = $this->getIdArchivesByMonth($doneFlags);
return $idArchivesByMonth;
}
@ -587,6 +685,8 @@ class Archive
*/
private function cacheArchiveIdsAfterLaunching($archiveGroups, $plugins)
{
$this->invalidatedReportsIfNeeded();
$today = Date::today();
foreach ($this->params->getPeriods() as $period) {
@ -600,14 +700,14 @@ class Archive
// we already know there are no stats for this period
// we add one day to make sure we don't miss the day of the website creation
if ($twoDaysAfterPeriod->isEarlier($site->getCreationDate())) {
Log::verbose("Archive site %s, %s (%s) skipped, archive is before the website was created.",
Log::debug("Archive site %s, %s (%s) skipped, archive is before the website was created.",
$idSite, $period->getLabel(), $period->getPrettyString());
continue;
}
// if the starting date is in the future we know there is no visiidsite = ?t
if ($twoDaysBeforePeriod->isLater($today)) {
Log::verbose("Archive site %s, %s (%s) skipped, archive is after today.",
Log::debug("Archive site %s, %s (%s) skipped, archive is after today.",
$idSite, $period->getLabel(), $period->getPrettyString());
continue;
}
@ -627,7 +727,7 @@ class Archive
private function cacheArchiveIdsWithoutLaunching($plugins)
{
$idarchivesByReport = ArchiveSelector::getArchiveIds(
$this->params->getIdSites(), $this->params->getPeriods(), $this->params->getSegment(), $plugins, $this->params->isSkipAggregationOfSubTables());
$this->params->getIdSites(), $this->params->getPeriods(), $this->params->getSegment(), $plugins);
// initialize archive ID cache for each report
foreach ($plugins as $plugin) {
@ -655,8 +755,7 @@ class Archive
$this->params->getIdSites(),
$this->params->getSegment(),
$this->getPeriodLabel(),
$plugin,
$this->params->isSkipAggregationOfSubTables()
$plugin
);
}
@ -707,11 +806,6 @@ class Archive
return round((float)$value, 2);
}
private function uncompress($data)
{
return @gzuncompress($data);
}
/**
* Initializes the archive ID cache ($this->idarchives) for a particular 'done' flag.
*
@ -723,6 +817,8 @@ class Archive
* If this function is not called, then periods with no visits will not add
* entries to the cache. If the archive is used again, SQL will be executed to
* try and find the archive IDs even though we know there are none.
*
* @param string $doneFlag
*/
private function initializeArchiveIdCache($doneFlag)
{
@ -745,7 +841,10 @@ class Archive
*/
private function getArchiveGroupOfPlugin($plugin)
{
if ($this->getPeriodLabel() != 'range') {
$periods = $this->params->getPeriods();
$periodLabel = reset($periods)->getLabel();
if (Rules::shouldProcessReportsAllPlugins($this->params->getIdSites(), $this->params->getSegment(), $periodLabel)) {
return self::ARCHIVE_ALL_PLUGINS_FLAG;
}
@ -755,7 +854,7 @@ class Archive
/**
* Returns the name of the plugin that archives a given report.
*
* @param string $report Archive data name, eg, `'nb_visits'`, `'UserSettings_...'`, etc.
* @param string $report Archive data name, eg, `'nb_visits'`, `'DevicesDetection_...'`, etc.
* @return string Plugin name.
* @throws \Exception If a plugin cannot be found or if the plugin for the report isn't
* activated.
@ -766,9 +865,9 @@ class Archive
if (in_array($report, Metrics::getVisitsMetricNames())) {
$report = 'VisitsSummary_CoreMetrics';
} // Goal_* metrics are processed by the Goals plugin (HACK)
else if (strpos($report, 'Goal_') === 0) {
elseif (strpos($report, 'Goal_') === 0) {
$report = 'Goals_Metrics';
} else if (strrpos($report, '_returning') === strlen($report) - strlen('_returning')) { // HACK
} elseif (strrpos($report, '_returning') === strlen($report) - strlen('_returning')) { // HACK
$report = 'VisitFrequency_Metrics';
}
@ -789,7 +888,7 @@ class Archive
*/
private function prepareArchive(array $archiveGroups, Site $site, Period $period)
{
$parameters = new ArchiveProcessor\Parameters($site, $period, $this->params->getSegment(), $this->params->isSkipAggregationOfSubTables());
$parameters = new ArchiveProcessor\Parameters($site, $period, $this->params->getSegment());
$archiveLoader = new ArchiveProcessor\Loader($parameters);
$periodString = $period->getRangeString();
@ -801,9 +900,37 @@ class Archive
$idArchive = $archiveLoader->prepareArchive($plugin);
if($idArchive) {
if ($idArchive) {
$this->idarchives[$doneFlag][$periodString][] = $idArchive;
}
}
}
private function getIdArchivesByMonth($doneFlags)
{
// order idarchives by the table month they belong to
$idArchivesByMonth = array();
foreach (array_keys($doneFlags) as $doneFlag) {
if (empty($this->idarchives[$doneFlag])) {
continue;
}
foreach ($this->idarchives[$doneFlag] as $dateRange => $idarchives) {
foreach ($idarchives as $id) {
$idArchivesByMonth[$dateRange][] = $id;
}
}
}
return $idArchivesByMonth;
}
/**
* @internal
*/
public static function clearStaticCache()
{
self::$cache = null;
}
}

317
www/analytics/core/Archive/ArchiveInvalidator.php Normal file
View file

@ -0,0 +1,317 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\Archive\ArchiveInvalidator\InvalidationResult;
use Piwik\CronArchive\SitesToReprocessDistributedList;
use Piwik\DataAccess\ArchiveTableCreator;
use Piwik\DataAccess\Model;
use Piwik\Date;
use Piwik\Option;
use Piwik\Plugins\CoreAdminHome\Tasks\ArchivesToPurgeDistributedList;
use Piwik\Plugins\PrivacyManager\PrivacyManager;
use Piwik\Period;
use Piwik\Segment;
/**
* Service that can be used to invalidate archives or add archive references to a list so they will
* be invalidated later.
*
* Archives are put in an "invalidated" state by setting the done flag to `ArchiveWriter::DONE_INVALIDATED`.
* This class also adds the archive's associated site to the a distributed list and adding the archive's year month to another
* distributed list.
*
* CronArchive will reprocess the archive data for all sites in the first list, and a scheduled task
* will purge the old, invalidated data in archive tables identified by the second list.
*
* Until CronArchive, or browser triggered archiving, re-processes data for an invalidated archive, the invalidated
* archive data will still be displayed in the UI and API.
*
* ### Deferred Invalidation
*
* Invalidating archives means running queries on one or more archive tables. In some situations, like during
* tracking, this is not desired. In such cases, archive references can be added to a list via the
* rememberToInvalidateArchivedReportsLater method, which will add the reference to a distributed list
*
* Later, during Piwik's normal execution, the list will be read and every archive it references will
* be invalidated.
*/
class ArchiveInvalidator
{
private $rememberArchivedReportIdStart = 'report_to_invalidate_';
/**
* @var Model
*/
private $model;
public function __construct(Model $model)
{
$this->model = $model;
}
public function rememberToInvalidateArchivedReportsLater($idSite, Date $date)
{
$key = $this->buildRememberArchivedReportId($idSite, $date->toString());
$value = Option::get($key);
// we do not really have to get the value first. we could simply always try to call set() and it would update or
// insert the record if needed but we do not want to lock the table (especially since there are still some
// MyISAM installations)
if (false === $value) {
Option::set($key, '1');
}
}
public function getRememberedArchivedReportsThatShouldBeInvalidated()
{
$reports = Option::getLike($this->rememberArchivedReportIdStart . '%_%');
$sitesPerDay = array();
foreach ($reports as $report => $value) {
$report = str_replace($this->rememberArchivedReportIdStart, '', $report);
$report = explode('_', $report);
$siteId = (int) $report[0];
$date = $report[1];
if (empty($sitesPerDay[$date])) {
$sitesPerDay[$date] = array();
}
$sitesPerDay[$date][] = $siteId;
}
return $sitesPerDay;
}
private function buildRememberArchivedReportId($idSite, $date)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite);
$id .= '_' . trim($date);
return $id;
}
private function buildRememberArchivedReportIdForSite($idSite)
{
return $this->rememberArchivedReportIdStart . (int) $idSite;
}
public function forgetRememberedArchivedReportsToInvalidateForSite($idSite)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite) . '_%';
Option::deleteLike($id);
}
/**
* @internal
*/
public function forgetRememberedArchivedReportsToInvalidate($idSite, Date $date)
{
$id = $this->buildRememberArchivedReportId($idSite, $date->toString());
Option::delete($id);
}
/**
* @param $idSites int[]
* @param $dates Date[]
* @param $period string
* @param $segment Segment
* @param bool $cascadeDown
* @return InvalidationResult
* @throws \Exception
*/
public function markArchivesAsInvalidated(array $idSites, array $dates, $period, Segment $segment = null, $cascadeDown = false)
{
$invalidationInfo = new InvalidationResult();
$datesToInvalidate = $this->removeDatesThatHaveBeenPurged($dates, $invalidationInfo);
if (empty($period)) {
// if the period is empty, we don't need to cascade in any way, since we'll remove all periods
$periodDates = $this->getDatesByYearMonthAndPeriodType($dates);
} else {
$periods = $this->getPeriodsToInvalidate($datesToInvalidate, $period, $cascadeDown);
$periodDates = $this->getPeriodDatesByYearMonthAndPeriodType($periods);
}
$periodDates = $this->getUniqueDates($periodDates);
$this->markArchivesInvalidated($idSites, $periodDates, $segment);
$yearMonths = array_keys($periodDates);
$this->markInvalidatedArchivesForReprocessAndPurge($idSites, $yearMonths);
foreach ($idSites as $idSite) {
foreach ($dates as $date) {
$this->forgetRememberedArchivedReportsToInvalidate($idSite, $date);
}
}
return $invalidationInfo;
}
/**
* @param string[][][] $periodDates
* @return string[][][]
*/
private function getUniqueDates($periodDates)
{
$result = array();
foreach ($periodDates as $yearMonth => $periodsByYearMonth) {
foreach ($periodsByYearMonth as $periodType => $periods) {
$result[$yearMonth][$periodType] = array_unique($periods);
}
}
return $result;
}
/**
* @param Date[] $dates
* @param string $periodType
* @param bool $cascadeDown
* @return Period[]
*/
private function getPeriodsToInvalidate($dates, $periodType, $cascadeDown)
{
$periodsToInvalidate = array();
foreach ($dates as $date) {
if ($periodType == 'range') {
$date = $date . ',' . $date;
}
$period = Period\Factory::build($periodType, $date);
$periodsToInvalidate[] = $period;
if ($cascadeDown) {
$periodsToInvalidate = array_merge($periodsToInvalidate, $period->getAllOverlappingChildPeriods());
}
if ($periodType != 'year'
&& $periodType != 'range'
) {
$periodsToInvalidate[] = Period\Factory::build('year', $date);
}
}
return $periodsToInvalidate;
}
/**
* @param Period[] $periods
* @return string[][][]
*/
private function getPeriodDatesByYearMonthAndPeriodType($periods)
{
$result = array();
foreach ($periods as $period) {
$date = $period->getDateStart();
$periodType = $period->getId();
$yearMonth = ArchiveTableCreator::getTableMonthFromDate($date);
$result[$yearMonth][$periodType][] = $date->toString();
}
return $result;
}
/**
* Called when deleting all periods.
*
* @param Date[] $dates
* @return string[][][]
*/
private function getDatesByYearMonthAndPeriodType($dates)
{
$result = array();
foreach ($dates as $date) {
$yearMonth = ArchiveTableCreator::getTableMonthFromDate($date);
$result[$yearMonth][null][] = $date->toString();
// since we're removing all periods, we must make sure to remove year periods as well.
// this means we have to make sure the january table is processed.
$janYearMonth = $date->toString('Y') . '_01';
$result[$janYearMonth][null][] = $date->toString();
}
return $result;
}
/**
* @param int[] $idSites
* @param string[][][] $dates
* @throws \Exception
*/
private function markArchivesInvalidated($idSites, $dates, Segment $segment = null)
{
$archiveNumericTables = ArchiveTableCreator::getTablesArchivesInstalled($type = ArchiveTableCreator::NUMERIC_TABLE);
foreach ($archiveNumericTables as $table) {
$tableDate = ArchiveTableCreator::getDateFromTableName($table);
if (empty($dates[$tableDate])) {
continue;
}
$this->model->updateArchiveAsInvalidated($table, $idSites, $dates[$tableDate], $segment);
}
}
/**
* @param Date[] $dates
* @param InvalidationResult $invalidationInfo
* @return \Piwik\Date[]
*/
private function removeDatesThatHaveBeenPurged($dates, InvalidationResult $invalidationInfo)
{
$this->findOlderDateWithLogs($invalidationInfo);
$result = array();
foreach ($dates as $date) {
// we should only delete reports for dates that are more recent than N days
if ($invalidationInfo->minimumDateWithLogs
&& $date->isEarlier($invalidationInfo->minimumDateWithLogs)
) {
$invalidationInfo->warningDates[] = $date->toString();
continue;
}
$result[] = $date;
$invalidationInfo->processedDates[] = $date->toString();
}
return $result;
}
private function findOlderDateWithLogs(InvalidationResult $info)
{
// If using the feature "Delete logs older than N days"...
$purgeDataSettings = PrivacyManager::getPurgeDataSettings();
$logsDeletedWhenOlderThanDays = (int)$purgeDataSettings['delete_logs_older_than'];
$logsDeleteEnabled = $purgeDataSettings['delete_logs_enable'];
if ($logsDeleteEnabled
&& $logsDeletedWhenOlderThanDays
) {
$info->minimumDateWithLogs = Date::factory('today')->subDay($logsDeletedWhenOlderThanDays);
}
}
/**
* @param array $idSites
* @param array $yearMonths
*/
private function markInvalidatedArchivesForReprocessAndPurge(array $idSites, $yearMonths)
{
$store = new SitesToReprocessDistributedList();
$store->add($idSites);
$archivesToPurge = new ArchivesToPurgeDistributedList();
$archivesToPurge->add($yearMonths);
}
}

56
www/analytics/core/Archive/ArchiveInvalidator/InvalidationResult.php Normal file
View file

@ -0,0 +1,56 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Archive\ArchiveInvalidator;
use Piwik\Date;
/**
* Information about the result of an archive invalidation operation.
*/
class InvalidationResult
{
/**
* Dates that couldn't be invalidated because they are earlier than the configured log
* deletion limit.
*
* @var array
*/
public $warningDates = array();
/**
* Dates that were successfully invalidated.
*
* @var array
*/
public $processedDates = array();
/**
* The day of the oldest log entry.
*
* @var Date|bool