coderkun/questlab
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Wiki Activity

update Piwik to version 2.16 (fixes #91)

Browse source
This commit is contained in:
oliver 2016-04-10 18:55:57 +02:00
commit 47263617c5
5833 changed files with 418860 additions and 226988 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.hgignore
View file

@ -6,5 +6,5 @@ syntax: regexp
^seminarymedia/*
^seminaryuploads/*
^www/analytics/config/config.ini.php*
^www/analytics/temp/*
^www/analytics/tmp/*
^app/lib/phpqrcode/cache/*

6
views/html/html.tpl
View file

@ -128,6 +128,7 @@
<?=$seminarybar?>
<?php endif ?>
</aside>
<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
_paq.push(['trackPageView']);
@ -136,11 +137,12 @@
var u=(("https:" == document.location.protocol) ? "https" : "http") + "://" + document.location.hostname + "/analytics/";
_paq.push(['setTrackerUrl', u+'piwik.php']);
_paq.push(['setSiteId', 1]);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.type='text/javascript';
g.defer=true; g.async=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script>
<noscript><p><img src="/analytics/piwik.php?idsite=1" style="border:0;" alt="" /></p></noscript>
<!-- End Piwik Code -->
</body>
</html>

411
www/analytics/CHANGELOG.md Normal file
View file

@ -0,0 +1,411 @@
# Piwik Platform Changelog
This is a changelog for Piwik platform developers. All changes for our HTTP API's, Plugins, Themes, etc will be listed here.
## Piwik 2.16.0
### New features
* New segment `actionType` lets you segment all actions of a given type, eg. `actionType==events` or `actionType==downloads`. Action types values are: `pageviews`, `contents`, `sitesearches`, `events`, `outlinks`, `downloads`
* New segment `actionUrl` lets you segment any action that matches a given URL, whether they are Pageviews, Site searches, Contents, Downloads or Events.
* New segment `deviceBrand` lets you restrict your users to those using a particular device brand such as Apple, Samsung, LG, Google, Nokia, Sony, Lenovo, Alcatel, etc. View the [complete list of device brands.](http://developer.piwik.org/api-reference/segmentation)
* New segment operators `=^` "Starts with" and `=$` "Ends with" complement the existing segment operators: Contains, Does not contain, Equals, Not equals, Greater than or equal to, Less than or equal to.
* The JavaScript Tracker method `PiwikTracker.setDomains()` can now handle paths. This means when setting eg `_paq.push(['setDomains, '*.piwik.org/website1'])` all link that goes to the same domain `piwik.org` but to any other path than `website1/*` will be treated as outlink.
* In Administration > Websites, for each website, there is a checkbox "Only track visits and actions when the action URL starts with one of the above URLs". In Piwik 2.14.0, any action URL starting with one of the Alias URLs or starting with a subdomain of the Alias URL would be tracked. As of Piwik 2.15.0, when this checkbox is enabled, it may track less data: action URLs on an Alias URL subdomain will not be tracked anymore (you must specify each sub-domain as Alias URL).
* It is now possible to pass an option `php-cli-options` to the `core:archive` command. The given cli options will be forwarded to the actual PHP command. This allows to for example specifiy a different memory limit for the archiving process like this: `./console core:archive --php-cli-options="-d memory_limit=8G"`
* New less variable `@theme-color-menu-contrast-textSelected` that lets you specify the color of a selected menu item.
* in Administration > Diagnostics, there is a new page `Config file` which lets Super User view all config values from `global.ini.php` in the UI, and whether they were overriden in your `config/config.ini.php`
### New commands
* New command `config:set` lets you set INI config options from the command line. This command can be used for convenience or for automation.
### Internal changes
* `UsersManager.*` API calls: when an API request specifies a `token_auth` of a user with `admin` permission, the returned dataset will not include all usernames as previously, API will now only return usernames for users with `view` or `admin` permission to website(s) viewable by this `token_auth`.
* When generating a new plugin skeleton via `generate:plugin` command, plugin name must now contain only letters and numbers.
* JavaScript Tracker tests no longer require `SQLite`. The existing MySQL configuration for tests is used now. In order to run the tests make sure Piwik is installed and `[database_tests]` is configured in `config/config.ini.php`.
* The definitions for search engine and social network detection have been moved from bundled data files to a separate package (see [https://github.com/piwik/searchengine-and-social-list](https://github.com/piwik/searchengine-and-social-list)).
* In [UI screenshot tests](https://developer.piwik.org/guides/tests-ui), a test environment `configOverride` setting should be no longer overwritten. Instead new values should be added to the existing `configOverride` array in PHP or JavaScript. For example instead of `testEnvironment.configOverride = {group: {name: 1}}` use `testEnvironment.overrideConfig('group', 'name', '1')`.
### New APIs
* Add your own SMS/Text provider by creating a new class in the `SMSProvider` directory of your plugin. The class has to extend `Piwik\Plugins\MobileMessaging\SMSProvider` and implement the required methods.
* Segments can now be composed by a union of multiple segments. To do this set an array of segments that shall be used for that segment `$segment->setUnionOfSegments(array('outlinkUrl', 'downloadUrl'))` instead of defining a SQL column.
### Deprecations
* The method `DB::tableExists` was un-used and has been removed.
## Piwik 2.15.0
### New commands
* New command `diagnostics:analyze-archive-table` that analyzes archive tables
* New command `database:optimize-archive-tables` to optimize archive tables and possibly save disk space (even if on InnoDB)
* New Command `core:invalidate-report-data` to invalidate archive data (w/ period cascading) ([FAQ](https://piwik.org/faq/how-to/faq_155/))
### New APIs and features
* Piwik 2.15.0 is now mostly compatible with PHP7.
* The JavaScript Tracker `piwik.js` got a new method `logAllContentBlocksOnPage` to log all found content blocks within a page to the console. This is useful to debug / test content tracking. It can be triggered via `_paq.push(['logAllContentBlocksOnPage'])`
* The Class `Piwik\Plugins\Login\Controller` is now considered a public API.
* The new method `Piwik\Menu\MenuAbstract::registerMenuIcon()` can be used to define an icon for a menu category to replace the default arrow icon.
* New event `CronArchive.getIdSitesNotUsingTracker` that allows you to set a list of idSites that do not use the Tracker API to make sure we archive these sites if needed.
* New events `CronArchive.init.start` which is triggered when the CLI archiver starts and `CronArchive.end` when the archiver ended.
* Piwik tracker can now be configured with strict Content Security Policy ([CSP FAQ](https://piwik.org/faq/general/faq_20904/)).
* Super Users can choose whether to use the latest stable release or latest Long Term Support release.
### Breaking Changes
* The method `Dimension::getId()` has been set as `final`. It is not allowed to overwrite this method.
* We fixed a bug where the API method `Sites.getPatternMatchSites` only returned a very limited number of websites by default. We now return all websites by default unless a limit is specified specifically.
* Handling of localized date, time and range formats has been changed. Patterns no longer contain placeholders like %shortDay%, but work with CLDR pattern instead. You can use one of the predefined format constants in Date class for using getLocalized().
* As we are now using CLDR formats for all languages, some time formats were even changed in english. Attributes like prettyDate in API responses might so have been changed slightly.
* The config `enable_measure_piwik_usage_in_idsite` which is used to track the Piwik usage with Piwik was removed and replaced by a new plugin `AnonymousPiwikUsageMeasurement`
### Deprecations
* The following HTTP API methods have been deprecated and will be removed in Piwik 3.0:
* `SitesManager.getSitesIdWithVisits`
* `API.getLastDate`
* The following events have been deprecated and will be removed in Piwik 3.0. Use [dimensions](http://developer.piwik.org/guides/dimensions) instead.
* `Tracker.existingVisitInformation`
* `Tracker.getVisitFieldsToPersist`
* `Tracker.newConversionInformation`
* `Tracker.newVisitorInformation`
* `Tracker.recordAction`
* `Tracker.recordEcommerceGoal`
* `Tracker.recordStandardGoals`
* The Platform API method `\Piwik\Plugin::getListHooksRegistered()` has been deprecated and will be removed in Piwik 3.0. Use `\Piwik\Plugin::registerEvents()` instead.
### Internal changes
* When logging in, the username is now case insensitive
* URLs with emojis and any other unicode character will be tracked, with special characters replaced with `<60>`
* A permanent warning notification is now displayed when PHP is 5.4.* or older, since it has reached End Of Life
* In `piwik.js` we replaced [JSON2](https://github.com/douglascrockford/JSON-js) with [JSON3](https://bestiejs.github.io/json3/) to implement CSP (Content Security Policy) as JSON3 does not use `eval()`. JSON3 will be used if a browser does not provide a native JSON API. We are using `JSON3` in a way that it will not conflict if your website is using `JSON3` as well.
* The option `branch` of the console command `development:sync-system-test-processed` was removed as it is no longer needed.
* All numbers in reports will now appear formatted (eg. `1,000,000` instead of `1000000`)
* Database connections now use `UTF-8` charset explicitely to force UTF-8 data handling
## Piwik 2.14.0
### Breaking Changes
* The `UserSettings` API has been removed. The API was deprecated in earlier versions. Use `DevicesDetection`, `Resolution` and `DevicePlugins` API instead.
* Many translations have been moved to the new Intl plugin. Most of them will still work, but please update their usage. See https://github.com/piwik/piwik/pull/8101 for a full list
### New features
* The JavaScript Tracker does now track outlinks and downloads if a user opens the context menu if the `enabled` parameter of the `enableLinkTracking()` method is set to `true`. To use this new feature use `tracker.enableLinkTracking(true)` or `_paq.push(['enableLinkTracking', true]);`. This is not industry standard and is vulnerable to false positives since not every user will select "Open in a new tab" when the context menu is shown. Most users will do though and it will lead to more accurate results in most cases.
* The JavaScript Tracker now contains the 'heart beat' feature which can be used to obtain more accurate visit lengths by periodically sending 'ping' requests to Piwik. To use this feature use `tracker.enableHeartBeatTimer();` or `_paq.push(['enableHeartBeatTimer']);`. By default, a ping request will be sent every 15 seconds. You can specify a custom ping delay (in seconds) by passing an argument, eg, `tracker.enableHeartBeatTimer(10);` or `_paq.push(['enableHeartBeatTimer', 10]);`.
* New custom segment `languageCode` that lets you segment visitors that are using a particular language. Example values: `de`, `fr`, `en-gb`, `zh-cn`, etc.
* Segment `userId` now supports any segment operator (previously only operator Contains `=@` was supported for this segment).
### Commands updates
* The command `core:archive` now has two new parameter: `--force-idsegments` and `--skip-idsegments` that let you force (or skip) processing archives for one or several custom segments.
* The command `scheduled-tasks:run` now has an argument `task` that lets you force run a particular scheduled task.
### Library updates
* Updated pChart library from 2.1.3 to 2.1.4. The files were moved from the directory `libs/pChart2.1.3` to `libs/pChart`
### Internal change
* To execute UI tests "ImageMagick" is now required.
* The Q JavaScript promise library is now distributed with tests and can be used in the piwik.js tests.
## Piwik 2.13.0
### Breaking Changes
* The API method `Live.getLastVisitsDetails` does no longer support the API parameter `filter_sort_column` to prevent possible memory issues when `filter_offset` is large.
* The Event `Site.setSite` was removed as it causes performance problems.
* `piwik.php` does now return a HTTP 400 (Bad request) if requested without any tracking parameters (GET/POST). If you still want to use `piwik.php` for checks please use `piwik.php?rec=0`.
### Deprecations
* The method `Piwik\Archive::getBlob()` has been deprecated and will be removed from June 1st 2015. Use one of the methods `getDataTable*()` methods instead.
* The API parameter `countVisitorsToFetch` of the API method `Live.getLastVisitsDetails` has been deprecated as `filter_offset` and `filter_limit` work correctly now.
### New commands
* There is now a `diagnostic:run` command to run the system check from the command line.
* There is now an option `--xhprof` that can be used with any command to profile that command via XHProf.
### APIs Improvements
* Visitor details now additionally contain: `deviceTypeIcon`, `deviceBrand` and `deviceModel`
* In 2.6.0 we added the possibility to use `filter_limit` and `filter_offset` if an API returns an indexed array. This was not working in all cases and is fixed now.
* The API parameter `filter_pattern` and `filter_offset[]` can now be used if an API returns an indexed array.
### Internal changes
* The referrer spam filter has moved from the `referrer_urls_spam` INI option (in `global.ini.php`) to a separate package (see [https://github.com/piwik/referrer-spam-blacklist](https://github.com/piwik/referrer-spam-blacklist)).
## Piwik 2.12.0
### Breaking Changes
* The deprecated method `Period::factory()` has been removed. Use `Period\Factory` instead.
* The deprecated method `Config::getConfigSuperUserForBackwardCompatibility()` has been removed.
* The deprecated methods `MenuAdmin::addEntry()` and `MenuAdmin::removeEntry()` have been removed. Use `Piwik\Plugin\Menu` instead.
* The deprecated methods `MenuTop::addEntry()` and `MenuTop::removeEntry()` have been removed. Use `Piwik\Plugin\Menu` instead.
* The deprecated method `SettingsPiwik::rewriteTmpPathWithInstanceId()` has been removed.
* The following deprecated methods from the `Piwik\IP` class have been removed, use `Piwik\Network\IP` instead:
* `sanitizeIp()`
* `sanitizeIpRange()`
* `P2N()`
* `N2P()`
* `prettyPrint()`
* `isIPv4()`
* `long2ip()`
* `isIPv6()`
* `isMappedIPv4()`
* `getIPv4FromMappedIPv6()`
* `getIpsForRange()`
* `isIpInRange()`
* `getHostByAddr()`
### Deprecations
* `API` classes should no longer have a protected constructor. Classes with a protected constructor will generate a notice in the logs and should expose a public constructor instead.
* Update classes should not declare static `getSql()` and `update()` methods anymore. It is still supported to use those, but developers should instead override the `Updates::getMigrationQueries()` and `Updates::doUpdate()` instance methods.
### New features
* `API` classes can now use dependency injection in their constructor to inject other instances.
### New commands
* There is now a command `core:purge-old-archive-data` that can be used to manually purge temporary, error-ed and invalidated archives from one or more archive tables.
* There is now a command `usercountry:attribute` that can be used to re-attribute geolocated location data to existing visits and conversions. If you have visits that were tracked before setting up GeoIP, you can use this command to add location data to them.
## Piwik 2.11.0
### Breaking Changes
* The event `User.getLanguage` has been removed.
* The following deprecated event has been removed: `TaskScheduler.getScheduledTasks`
* Special handling for operating system `Windows` has been removed. Like other operating systems all versions will now only be reported as `Windows` with versions like `XP`, `7`, `8`, etc.
* Reporting for operating systems has been adjusted to report information according to browser information. Visitor details now contain: `operatingSystemName`, `operatingSystemIcon`, `operatingSystemCode` and `operatingSystemVersion`
### Deprecations
* The following methods have been deprecated in favor of the new `Piwik\Intl` component:
* `Piwik\Common::getContinentsList()`: use `RegionDataProvider::getContinentList()` instead
* `Piwik\Common::getCountriesList()`: use `RegionDataProvider::getCountryList()` instead
* `Piwik\Common::getLanguagesList()`: use `LanguageDataProvider::getLanguageList()` instead
* `Piwik\Common::getLanguageToCountryList()`: use `LanguageDataProvider::getLanguageToCountryList()` instead
* `Piwik\Metrics\Formatter::getCurrencyList()`: use `CurrencyDataProvider::getCurrencyList()` instead
* The `Piwik\Translate` class has been deprecated in favor of `Piwik\Translation\Translator`.
* The `core:plugin` console has been deprecated in favor of the new `plugin:list`, `plugin:activate` and `plugin:deactivate` commands
* The following classes have been deprecated:
* `Piwik\TaskScheduler`: use `Piwik\Scheduler\Scheduler` instead
* `Piwik\ScheduledTask`: use `Piwik\Scheduler\Task` instead
* The API method `UserSettings.getLanguage` is deprecated and will be removed from May 1st 2015. Use `UserLanguage.getLanguage` instead
* The API method `UserSettings.getLanguageCode` is deprecated and will be removed from May 1st 2015. Use `UserLanguage.getLanguageCode` instead
* The `Piwik\Registry` class has been deprecated in favor of using the container:
* `Registry::get('auth')` should be replaced with `StaticContainer::get('Piwik\Auth')`
* `Registry::set('auth', $auth)` should be replaced with `StaticContainer::getContainer()->set('Piwik\Auth', $auth)`
### New features
* You can now generate UI / screenshot tests using the command `generate:test`
* During UI tests we do now add a CSS class to the HTML element called `uiTest`. This allows you do hide content when screenshots are captured.
### New commands
* A new command (core:fix-duplicate-log-actions) has been added which can be used to remove duplicate actions and correct references to them in other tables. Duplicates were caused by this bug: [#6436](https://github.com/piwik/piwik/issues/6436)
### Library updates
* Updated AngularJS from 1.2.26 to 1.2.28
* Updated piwik/device-detector from 2.8 to 3.0
### Internal change
* UI specs were moved from `tests/PHPUnit/UI` to `tests/UI`. We also moved the UI specs directly into the Piwik repository meaning the [piwik-ui-tests](https://github.com/piwik/piwik-ui-tests) repository contains only the expected screenshots from now on.
* There is a new command `development:sync-system-test-processed` for core developers that allows you to copy processed test results from travis to your local dev environment.
## Piwik 2.10.0
### Breaking Changes
* API responses containing visitor information will no longer contain the fields `screenType` and `screenTypeIcon` as those reports have been completely removed
* os, browser and browser plugin icons are now located in the DevicesDetection and DevicePlugins plugin. If you are not using the Reporting or Metadata API to get the icon locations please update your paths.
* The deprecated method `Piwik\SettingsPiwik::rewriteTmpPathWithHostname()` has been removed.
* The following events have been removed:
* `Log.formatFileMessage`
* `Log.formatDatabaseMessage`
* `Log.formatScreenMessage`
* These events have been removed as Piwik now uses the Monolog logging library. [Learn more.](http://developer.piwik.org/guides/logging)
* The event `Log.getAvailableWriters` has been removed: to add custom log backends, you now need to configure Monolog handlers
* The INI options `log_only_when_cli` and `log_only_when_debug_parameter` have been removed
### Library updates
* We added the `symfony/var-dumper` library allowing you to better print any arbitrary PHP variable via `dump($var1, $var2, ...)`.
* Piwik now uses [Monolog](https://github.com/Seldaek/monolog) as a logger.
* The tracker proxy (previously in `misc/proxy-hide-piwik-url/`) has been moved to a separate repository: [https://github.com/piwik/tracker-proxy](https://github.com/piwik/tracker-proxy).
### Deprecations
* Some duplicate reports from UserSettings plugin have been removed. Widget URLs for those reports will still work till May 1st 2015. Please update those to the new reports of DevicesDetection plugin.
* The API method `UserSettings.getBrowserVersion` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowserVersions` instead
* The API method `UserSettings.getBrowser` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowsers` instead
* The API method `UserSettings.getOSFamily` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getOsFamilies` instead
* The API method `UserSettings.getOS` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getOsVersions` instead
* The API method `UserSettings.getMobileVsDesktop` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getType` instead
* The API method `UserSettings.getBrowserType` is deprecated and will be removed from May 1st 2015. Use `DevicesDetection.getBrowserEngines` instead
* The API method `UserSettings.getResolution` is deprecated and will be removed from May 1st 2015. Use `Resolution.getResolution` instead
* The API method `UserSettings.getConfiguration` is deprecated and will be removed from May 1st 2015. Use `Resolution.getConfiguration` instead
* The API method `UserSettings.getPlugin` is deprecated and will be removed from May 1st 2015. Use `DevicePlugins.getPlugin` instead
* The API method `UserSettings.getWideScreen` has been removed. Use `UserSettings.getScreenType` instead.
* `Piwik\SettingsPiwik::rewriteTmpPathWithInstanceId()` has been deprecated. Instead of hardcoding the `tmp/` path everywhere in the codebase and then calling `rewriteTmpPathWithInstanceId()`, developers should get the `path.tmp` configuration value from the DI container (e.g. `StaticContainer::getContainer()->get('path.tmp')`).
* The method `Piwik\Log::setLogLevel()` has been deprecated
* The method `Piwik\Log::getLogLevel()` has been deprecated
## Piwik 2.9.1
### Breaking Changes
* The HTTP Tracker API does now respond with a HTTP 400 instead of a HTTP 500 in case an invalid `idsite` is used
### New APIs
* New URL parameter `send_image=0` in the [HTTP Tracking API](http://developer.piwik.org/api-reference/tracking-api) to receive a HTTP 204 response code instead of a GIF image. This improves performance and can fix errors if images are not allowed to be obtained directly (eg Chrome Apps).
### New commands
* `core:plugin list` lists all plugins currently activated in Piwik.
## Piwik 2.9.0
### Breaking Changes
* Development related [console commands](http://developer.piwik.org/guides/piwik-on-the-command-line) are only available if the development mode is enabled. To enable the development mode execute `./console development:enable`.
* The command `php console core:update` does no longer have a parameter `--dry-run`. A dry run is now executed by default followed by a question whether one actually wants to execute the updates. To skip this confirmation step one can use the `--yes` option.
### Deprecations
* Most methods of `Piwik\IP` have been deprecated in favor of the new [piwik/network](https://github.com/piwik/component-network) component.
* The file `tests/PHPUnit/phpunit.xml` is no longer needed in order to run tests and we suggest to delete it. The test configuration is now done automatically if possible. In case the tests do no longer work check out the `[tests]` section in `config/global.ini.php`
### Library updates
* Code for manipulating IP addresses has been moved to a separate standalone component: [piwik/network](https://github.com/piwik/component-network). Backward compatibility is kept in Piwik core.
## Piwik 2.8.2
### Library updates
* Updated AngularJS from 1.2.25 to 1.2.26
* Updated jQuery from 1.11.0 to 1.11.1
## Piwik 2.8.0
### Breaking Changes
* The Auth interface has been modified, existing Auth implementations will have to be modified. Changes include:
* The initSession method has been moved. Since this behavior must be executed for every Auth implementation, it has been put into a new class: SessionInitializer.
If your Auth implementation implements its own session logic you will have to extend and override SessionInitializer.
* The following methods have been added: setPassword, setPasswordHash, getTokenAuthSecret and getLogin.
* Clarifying semantics of each method and what they must support and can support.
* **Read the documentation for the [Auth interface](http://developer.piwik.org/api-reference/Piwik/Auth) to learn more.**
* The `Piwik\Unzip\*` classes have been extracted out of the Piwik repository into a separate component named [Decompress](https://github.com/piwik/component-decompress).
* `Piwik\Unzip` has not moved, it is kept for backward compatibility. If you have been using that class, you don't need to change anything.
* The `Piwik\Unzip\*` classes (Tar, PclZip, Gzip, ZipArchive) have moved to the `Piwik\Decompress\*` namespace (inside the new repository).
* `Piwik\Unzip\UncompressInterface` has been moved and renamed to `Piwik\Decompress\DecompressInterface` (inside the new repository).
### Deprecations
* The `Piwik::setUserHasSuperUserAccess` method is deprecated, instead use Access::doAsSuperUser. This method will ensure that super user access is properly rescinded after the callback finishes.
* The class `\IntegrationTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\SystemTestCase` instead.
* The class `\DatabaseTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\IntegrationTestCase` instead.
* The class `\BenchmarkTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\BenchmarkTestCase` instead.
* The class `\ConsoleCommandTestCase` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\TestCase\ConsoleCommandTestCase` instead.
* The class `\FakeAccess` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\Mock\FakeAccess` instead.
* The class `\Piwik\Tests\Fixture` is deprecated and will be removed from February 6th 2015. Use `\Piwik\Tests\Framework\Fixture` instead.
* The class `\Piwik\Tests\OverrideLogin` is deprecated and will be removed from February 6ths 2015. Use `\Piwik\Framework\Framework\OverrideLogin` instead.
### New API Features
* The pivotBy and related query parameters can be used to pivot reports by another dimension. Read more about the new query parameters [here](http://developer.piwik.org/api-reference/reporting-api#optional-api-parameters).
### Library updates
* Updated AngularJS from 1.2.13 to 1.2.25
### New commands
* `generate:angular-directive` Let's you easily generate a template for a new angular directive for any plugin.
### Internal change
* Piwik 2.8.0 now requires PHP >= 5.3.3.
* If you use an older PHP version, please upgrade now to the latest PHP so you can enjoy improvements and security fixes in Piwik.
## Piwik 2.7.0
### Reporting APIs
* Several APIs will now expose a new metric `nb_users` which measures the number of unique users when a [User ID](http://piwik.org/docs/user-id/) is set.
* New APIs have been added for [Content Tracking](http://piwik.org/docs/content-tracking/) feature: Contents.getContentNames, Contents.getContentPieces
### Deprecations
* The `Piwik\Menu\MenuAbstract::add()` method is deprecated in favor of `addItem()`. Read more about this here: [#6140](https://github.com/piwik/piwik/issues/6140). We do not plan to remove the deprecated method before Piwik 3.0.
### New APIs
* It is now easier to generate the URL for a menu item see [#6140](https://github.com/piwik/piwik/issues/6140), [urlForDefaultAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlfordefaultaction), [urlForAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlforaction), [urlForModuleAction()](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu#urlformoduleaction)
### New commands
* `core:clear-caches` Lets you easily delete all caches. This command can be useful for instance after updating Piwik files manually.
## Piwik 2.6.0
### Deprecations
* The `'json'` API format is considered deprecated. We ask all new code to use the `'json2'` format. Eventually when Piwik 3.0 is released the `'json'` format will be replaced with `'json2'`. Differences in the json2 format include:
* A bug in JSON formatting was fixed so API methods that return simple associative arrays like `array('name' => 'value', 'name2' => 'value2')` will now appear correctly as `{"name":"value","name2":"value2"}` in JSON API output instead of `[{"name":"value","name2":"value2"}]`. API methods like **SitesManager.getSiteFromId** & **UsersManager.getUser** are affected.
#### Reporting API
* If an API returns an indexed array, it is now possible to use `filter_limit` and `filter_offset`. This was before only possible if an API returned a DataTable.
* The Live API now returns only visitor information of activated plugins. So if for instance the Referrers plugin is deactivated a visitor won't contain any referrers related properties. This is a bugfix as the API was crashing before if some core plugins were deactivated. Affected methods are for instance `getLastVisitDetails` or `getVisitorProfile`. If all core plugins are enabled as by default there will be no change at all except the order of the properties within one visitor.
### New commands
* `core:run-scheduled-tasks` Let's you run all scheduled tasks due to run at this time. Useful for instance when testing tasks.
#### Internal change
* We removed our own autoloader that was used to load Piwik files in favor of the composer autoloader which we already have been using for some libraries. This means the file `core/Loader.php` will no longer exist. In case you are using Piwik from Git make sure to run `php composer.phar self-update && php composer.phar install` to make your Piwik work again. Also make sure to no longer include `core/Loader.php` in case it is used in any custom script.
* We do no longer store the list of plugins that are used during tracking in the config file. They are dynamically detect instead. The detection of a tracker plugin works the same as before. A plugin has to either listen to any `Tracker.*` or `Request.initAuthenticationObject` event or it has to define dimensions in order to be detected as a tracker plugin.
## Piwik 2.5.0
### Breaking Changes
* Javascript Tracking API: if you are using `getCustomVariable` function to access custom variables values that were set on previous page views, you now must also call `storeCustomVariablesInCookie` before the first call to `trackPageView`. Read more about [Javascript Tracking here](http://developer.piwik.org/api-reference/tracking-javascript).
* The [settings](http://developer.piwik.org/guides/piwik-configuration) API will receive the actual entered value and will no longer convert characters like `&` to `&amp;`. If you still want this behavior - for instance to prevent XSS - you can define a filter by setting the `transform` property like this:
`$setting->transform = function ($value) { return Common::sanitizeInputValue($value); }`
* Config setting `disable_merged_assets` moved from `Debug` section to `Development`. The updater will automatically change the section for you.
* `API.getRowEvolution` will throw an exception if a report is requested that does not have a dimension, for instance `VisitsSummary.get`. This is a fix as an invalid format was returned before see [#5951](https://github.com/piwik/piwik/issues/5951)
* `MultiSites.getAll` returns from now on always an array of websites. In the past it returned a single object and it didn't contain all properties in case only one website was found which was a bug see [#5987](https://github.com/piwik/piwik/issues/5987)
### Deprecations
The following events are considered as deprecated and the new structure should be used in the future. We have not scheduled when those events will be removed but probably in Piwik 3.0 which is not scheduled yet and won't be soon. New features will be added only to the new classes.
* `API.getReportMetadata`, `API.getSegmentDimensionMetadata`, `Goals.getReportsWithGoalMetrics`, `ViewDataTable.configure`, `ViewDataTable.getDefaultType`: use [Report](http://developer.piwik.org/api-reference/Piwik/Plugin/Report) class instead to define new reports. There is an updated guide as well [Part1](http://developer.piwik.org/guides/getting-started-part-1)
* `WidgetsList.addWidgets`: use [Widgets](http://developer.piwik.org/api-reference/Piwik/Plugin/Widgets) class instead to define new widgets
* `Menu.Admin.addItems`, `Menu.Reporting.addItems`, `Menu.Top.addItems`: use [Menu](http://developer.piwik.org/api-reference/Piwik/Plugin/Menu) class instead
* `TaskScheduler.getScheduledTasks`: use [Tasks](http://developer.piwik.org/api-reference/Piwik/Plugin/Tasks) class instead to define new tasks
* `Tracker.recordEcommerceGoal`, `Tracker.recordStandardGoals`, `Tracker.newConversionInformation`: use [Conversion Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ConversionDimension) class instead
* `Tracker.existingVisitInformation`, `Tracker.newVisitorInformation`, `Tracker.getVisitFieldsToPersist`: use [Visit Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/VisitDimension) class instead
* `ViewDataTable.addViewDataTable`: This event is no longer needed. Visualizations are automatically discovered if they are placed within a `Visualizations` directory inside the plugin.
### New features
#### Translation search
As a plugin developer you might want to reuse existing translation keys. You can now find all available translations and translation keys by opening the page "Settings => Development:Translation search" in your Piwik installation. Read more about [internationalization](http://developer.piwik.org/guides/internationalization) here.
#### Reporting API
It is now possible to use the `filter_sort_column` parameter when requesting `Live.getLastVisitDetails`. For instance `&filter_sort_column=visitCount`.
#### @since annotation
We are using `@since` annotations in case we are introducing new API's to make it easy to see in which Piwik version a new method was added. This information is now displayed in the [Classes API-Reference](http://developer.piwik.org/api-reference/classes).
### New APIs
* [Report](http://developer.piwik.org/api-reference/Piwik/Plugin/Report) to add a new report
* [Action Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ActionDimension) to add a dimension that tracks action related information
* [Visit Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/VisitDimension) to add a dimension that tracks visit related information
* [Conversion Dimension](http://developer.piwik.org/api-reference/Piwik/Plugin/Dimension/ConversionDimension) to add a dimension that tracks conversion related information
* [Dimension](http://developer.piwik.org/api-reference/Piwik/Columns/Dimension) to add a basic non tracking dimension that can be used in `Reports`
* [Widgets](http://developer.piwik.org/api-reference/Piwik/Plugin/Widgets) to add or modfiy widgets
* These Menu classes got new methods that make it easier to add new items to a specific section
* [MenuAdmin](http://developer.piwik.org/api-reference/Piwik/Menu/MenuAdmin) to add or modify admin menu items.
* [MenuReporting](http://developer.piwik.org/api-reference/Piwik/Menu/MenuReporting) to add or modify reporting menu items
* [MenuUser](http://developer.piwik.org/api-reference/Piwik/Menu/MenuUser) to add or modify user menu items
* [Tasks](http://developer.piwik.org/api-reference/Piwik/Plugin/Tasks) to add scheduled tasks
### New commands
* `generate:theme` Let's you easily generate a new theme and customize colors, see the [Theming guide](http://developer.piwik.org/guides/theming)
* `generate:update` Let's you generate an update file
* `generate:report` Let's you generate a report
* `generate:dimension` Let's you enhance the tracking by adding new dimensions
* `generate:menu` Let's you generate a menu class to add or modify menu items
* `generate:widgets` Let's you generate a widgets class to add or modify widgets
* `generate:tasks` Let's you generate a tasks class to add or modify tasks
* `development:enable` Let's you enable the development mode which will will disable some caching to make code changes directly visible and it will assist developers by performing additional checks to prevent for instance typos. Should not be used in production.
* `development:disable` Let's you disable the development mode
<!--
## Template: Piwik version number
### Breaking Changes
### Deprecations
### New features
### New APIs
### New commands
### New guides
### Library updates
### Internal change
-->
Find the general Piwik Changelogs for each release at [piwik.org/changelog](http://piwik.org/changelog/)

12
www/analytics/CONTRIBUTING.md Normal file
View file

@ -0,0 +1,12 @@
# How to contribute
Great to have you here! Read the following guide on our developer zone to learn how you can help make this project better!
http://developer.piwik.org/guides/contributing-to-piwik-core
## How to submit a bug report or suggest a feature?
Please read the recommendations on writing a good [bug report](http://developer.piwik.org/guides/core-team-workflow#submitting-a-bug-report) or [feature request](http://developer.piwik.org/guides/core-team-workflow#submitting-a-feature-request).
## How to suggest improvements to translations?
You can help improve translations in Piwik, please read [contribute to translations](https://github.com/piwik/piwik/blob/master/lang/README.md).

71
www/analytics/LEGALNOTICE
View file

@ -1,10 +1,10 @@
COPYRIGHT
Piwik - Open Source Web Analytics
Piwik - free/libre analytics platform
The software package is:
Copyright (C) 2013 Matthieu Aubry
Copyright (C) 2014 Matthieu Aubry
Individual contributions, components, and libraries are copyright
of their respective authors.
@ -40,7 +40,7 @@ CREDITS
For detailed contribution history, refer to the source, tickets,
patches, and Git revision history, available at
http://dev.piwik.org/trac/
https://github.com/piwik/piwik/issues
https://github.com/piwik/piwik
@ -65,9 +65,17 @@ SEPARATELY LICENSED COMPONENTS AND LIBRARIES
Link: https://github.com/piwik/piwik/blob/master/libs/PiwikTracker/
License: New BSD
Name: UserAgentParser
Link: https://github.com/piwik/piwik/blob/master/libs/UserAgentParser/
License: New BSD
Name: DeviceDetector
Link: https://github.com/piwik/device-detector
License: LGPL
Name: Piwik/Decompress
Link: https://github.com/piwik/component-decompress
License: LGPL v3.0
Name: Piwik/Network
Link: https://github.com/piwik/component-network
License: LGPL v3.0
THIRD-PARTY COMPONENTS AND LIBRARIES
@ -77,40 +85,40 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: jqPlot
Link: http://www.jqplot.com/
License: Dual-licensed: MIT or GPL v2
License: Dual-licensed: MIT (Expat) or GPL v2
Name: jQuery
Link: http://jquery.com/
License: Dual-licensed: MIT or GPL
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
- includes Sizzle.js - multi-licensed: MIT, New BSD, or GPL [v2]
- includes Sizzle.js - multi-licensed: MIT (Expat), New BSD, or GPL [v2]
Name: jQuery UI
Link: http://jqueryui.com/
License: Dual-licensed: MIT or GPL
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
Name: jquery.history
Link: http://tkyk.github.com/jquery-history-plugin/
License: MIT
License: MIT (Expat)
Name: jquery.scrollTo
Link: http://plugins.jquery.com/project/ScrollTo
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery Tooltip
Link: http://bassistance.de/jquery-plugins/jquery-plugin-tooltip/
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery placeholder
Link: http://mths.be/placeholder
License: Dual licensed: MIT or GPL
License: Dual licensed: MIT (Expat) or GPL
Name: jquery smartbanner
Link: https://github.com/jasny/jquery.smartbanner
License: Dual licensed: MIT
License: Dual licensed: MIT (Expat)
Name: json2.js
Link: http://json.org/
@ -187,8 +195,8 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: Zend Framework
Link: http://www.zendframework.com/
License: New BSD
Name: pChart 2.1.3
Name: pChart 2.1.4
Link: http://www.pchart.net
License: GPL v3
@ -206,15 +214,27 @@ THIRD-PARTY COMPONENTS AND LIBRARIES
Name: Raphaël - JavaScript Vector Library
Link: http://raphaeljs.com/
License: MIT
License: MIT (Expat)
Name: lessphp
Link: http://leafo.net/lessphp
License: GPL3/MIT
License: GPL3, MIT (Expat)
Name: Symfony Console Component
Link: https://github.com/symfony/Console
License: MIT
License: MIT (Expat)
Name: AngularJS
Link: https://github.com/angular/angular.js
License: MIT (Expat)
Name: Mousetrap
Link: https://github.com/ccampbell/mousetrap
License: Apache 2.0
Name: PHP-DI
Link: http://php-di.org/
License: MIT (Expat)
THIRD-PARTY CONTENT
@ -235,10 +255,6 @@ THIRD-PARTY CONTENT
Notes:
- used in ImageGraph plugin
Name: plugins/CorePluginsAdmin/images/themes.png
Link: https://www.iconfinder.com/icons/17022/colors_draw_paint_icon
License: Free for commercial use
Name: plugins/Feedback/angularjs/ratefeature/thumbs-down.png
Link: https://www.iconfinder.com/icons/216428/down_thumbs_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
@ -247,10 +263,6 @@ THIRD-PARTY CONTENT
Link: https://www.iconfinder.com/icons/216429/thumbs_up_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
Name: plugins/CorePluginsAdmin/images/plugins.png
Link: http://findicons.com/icon/94051/tools_wizard?id=396912
License: GNU/GPL
Name: plugins/Insights/images/idea.png
Link: https://www.iconfinder.com/icons/6074/brainstorm_bulb_idea_jabber_light_icon
License: GPL
@ -259,7 +271,6 @@ THIRD-PARTY CONTENT
Notes:
- the "New BSD" license refers to either the "Modified BSD" and "Simplified BSD"
licenses (2- or 3-clause), which are GPL compatible.
- the "MIT" license is also referred to as the "X11" license
- icons for browsers, operating systems, browser plugins, search engines, and
and flags of countries are nominative use of third-party trademarks when
referring to the corresponding product or entity

60
www/analytics/PRIVACY.md Normal file
View file

@ -0,0 +1,60 @@
# Privacy
This is a summary of all of the components within Piwik which may affect your privacy in some way. Please keep in mind
third party Themes, Plugins or Apps may introduce privacy concerns not listed here.
## Privacy for users being tracked by Piwik
In this section we document how to protect the privacy of visitors who are tracked by your Piwik analytics service.
### Anonymise visitor IP addresses
By default, Piwik stores the visitor IP address (IPv4 or IPv6 format) in the database for each new visitor.
If a visitor has a static IP address this means her browsing history can be easily identified across several days and
even across several websites tracked within the same Piwik server. You can anonymize IP addresses to ensure visitors cannot
be tracked this way: [How to anonymise IP addresses.](http://piwik.org/docs/privacy/#step-1-automatically-anonymize-visitor-ips)
### Delete old visitors logs
By default, Piwik stores tracked data forever. To better respect the privacy of your users, it is recommended to regularly
purge old data. You can configure Piwik to automatically delete log data older than a specified number of months:
[How to delete old visitors log data.](http://piwik.org/docs/privacy/#step-2-delete-old-visitors-logs)
### Include a tracking Opt-Out feature on your site
In your website, we recommended providing an easy way for your visitors to “opt-out” of being tracked by Piwik.
You can use the Opt-Out feature to display a link your website that sets a special browser cookie (`piwik_ignore`) when
clicked. Visitors that click that link will be ignored by Piwik in the future:
[How to include a tracking opt-out iframe.](http://piwik.org/docs/privacy/#step-3-include-a-web-analytics-opt-out-feature-on-your-site-using-an-iframe)
### Respect DoNotTrack preference
Do Not Track is a browser-level technology and policy proposal that lets visitors opt out of tracking by websites they
do not visit. Visitors can enable this preference in their browser, and then it's up to Piwik to respect it. By default,
Piwik is configured to ignore visitors that have enabled it:
[How to check if your Piwik respects DoNotTrack.] (http://piwik.org/docs/privacy/#step-4-respect-donottrack-preference)
### Disable tracking cookies
A cookie is a collection of information that a website stores on a visitor’s computer and accesses each time the visitor
returns. By default, Piwik uses cookies to aid in tracking visitor behavior. If someone gains access to a visitor's
computer, they could learn a few things about how the visitor visited your website. For many websites, this isn't a
problem, but for others where a strong level of privacy is required (like online banking), disabling tracking cookies may
be a good idea: [How to disable tracking cookies.](http://piwik.org/faq/general/faq_157/)
### Keep your visitors details private
Any user that has at least `view` access (the default access level) to Piwik can view detailed information for all users
tracked in Piwik (such as their IP addresses, visitor IDs, details of all past visits and actions, etc.) through features
provided by the `Live` plugin (such as the Visitor Log and Visitor Profile). As the Piwik administrator, you may decide
that not all of your users need access to this data. You can deactivate the `Live` plugin to prevent users from viewing
visitor details in the Administration > Plugins page.
## Privacy for Piwik admins and website owners
In this section we document how a Piwik administrator can better protect their own privacy.
### Keep your Piwik server URL private
By default, the Piwik Javascript code on all tracked websites contains the Piwik server URL. In some cases you might
want to hide this Piwik URL completely while still tracking all websites in your Piwik instance. To hide your Piwik
server's URL, you can modify the Javascript Tracking code and point it to a proxy piwik.php script instead of your actual
Piwik server: [How to keep Piwik server URL private.](http://piwik.org/faq/how-to/faq_132/)
### Automatic update check
From time to time, Piwik uses `api.piwik.org` to check if the current version of Piwik is the latest version of Piwik.
If an update is available, a notification is displayed allowing you to upgrade Piwik. To disable the update check,
and stop your instance from sending HTTP requests to `api.piwik.org`, deactivate the "Automatic update" feature by
setting `enable_auto_update = 0` in your configuration file `config/config.ini.php`.
Learn more about [Privacy in Piwik](http://piwik.org/privacy/).

50
www/analytics/README.md
View file

@ -1,14 +1,27 @@
# Piwik - piwik.org
# Piwik - piwik.org
[![Latest Stable Version](https://poser.pugx.org/piwik/piwik/v/stable)](https://packagist.org/packages/piwik/piwik)
[![Latest Unstable Version](https://poser.pugx.org/piwik/piwik/v/unstable)](https://packagist.org/packages/piwik/piwik)
[![Total Downloads](https://poser.pugx.org/piwik/piwik/downloads)](https://packagist.org/packages/piwik/piwik)
[![License](https://poser.pugx.org/piwik/piwik/license)](https://packagist.org/packages/piwik/piwik)
## Code Status
[![Build Status](https://travis-ci.org/piwik/piwik.svg?branch=master)](https://travis-ci.org/piwik/piwik)
[![Scrutinizer Code Quality](https://img.shields.io/scrutinizer/g/piwik/piwik.svg)](https://scrutinizer-ci.com/g/piwik/piwik?branch=master)
[![Code Coverage](https://scrutinizer-ci.com/g/piwik/piwik/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/piwik/piwik/?branch=master)
[![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/piwik/piwik.svg)](http://isitmaintained.com/project/piwik/piwik "Average time to resolve an issue")
[![Percentage of issues still open](http://isitmaintained.com/badge/open/piwik/piwik.svg)](http://isitmaintained.com/project/piwik/piwik "Percentage of issues still open")
## Description
Piwik is the leading Free/Libre open source Web Analytics platform.
Piwik is the leading Free/Libre open analytics platform.
Piwik is a full featured PHP MySQL software program that you download and install on your own webserver.
At the end of the five minute installation process you will be given a JavaScript code.
Simply copy and paste this tag on websites you wish to track and access your analytics reports in real time.
Piwik aims to be a Free software alternative to Google Analytics, and is already used on more than 1,000,000 websites.
Piwik aims to be a Free software alternative to Google Analytics, and is already used on more than 1,000,000 websites. Privacy is built-in!
## Mission Statement
@ -21,17 +34,27 @@ Or in short:
Piwik is released under the GPL v3 (or later) license, see [misc/gpl-3.0.txt](misc/gpl-3.0.txt)
## We’re seeking a talented Software Engineer
Are you looking for a new challenge? We are currently seeking a software engineer or software developer who is passionate about data processing, security, privacy, the open source and free/libre philosophy and usable interface design.
[View Job Description](https://piwik.org/blog/2015/01/piwik-expanding-seeking-talented-software-engineer-new-zealand-poland/) - [Apply online](http://piwik.org/jobs/)
This is for a full time position to work on the open source Piwik platform, either remotely or we can help the right candidate relocate to beautiful New Zealand (Wellington) or Poland (Wroclaw).
We are grateful if you can share the job description with your friends and wider network!
## Requirements
* PHP 5.3.2 or greater
* PHP 5.3.3 or greater
* MySQL 4.1 or greater, and either MySQLi or PDO library must be enabled
* Piwik is OS / server independent
See http://piwik.org/docs/requirements/
## Install
## Install
* Upload piwik to your webserver
* Upload piwik to your webserver
* Point your browser to the directory
* Follow the steps
* Add the given javascript code to your pages
@ -43,7 +66,7 @@ If you do not have a server, consider our Piwik Hosting partner: http://piwik.or
## Changelog
For the list of all tickets closed in the current and past releases, see http://piwik.org/changelog/
For the list of all tickets closed in the current and past releases, see http://piwik.org/changelog/. For the list of technical changes in the Piwik platform, see [http://developer.piwik.org/changelog](http://developer.piwik.org/changelog).
## Participate!
@ -64,7 +87,7 @@ About us: http://piwik.org/the-piwik-team/
What makes Piwik unique from the competition:
* Real time web analytics reports: in Piwik, reports are by default generated in real time.
* Real time web analytics reports: in Piwik, reports are by default generated in real time.
For high traffic websites, you can choose the frequency for reports to be processed.
* You own your web analytics data: since Piwik is installed on your server, the data is stored in your own database and you can get all the statistics
@ -74,19 +97,12 @@ What makes Piwik unique from the competition:
* Modern, easy to use User Interface: you can fully customize your dashboard, drag and drop widgets and more.
* Piwik features are built inside plugins: you can add new features and remove the ones you don’t need.
* Piwik features are built inside plugins: you can add new features and remove the ones you don’t need.
You can build your own web analytics plugins or hire a consultant to have your custom feature built in Piwik
* Vibrant international Open community of more than 200,000 active users (tracking even more websites!)
* Advanced Web Analytics capabilities such as Ecommerce Tracking, Goal tracking, Campaign tracking,
* Advanced Web Analytics capabilities such as Ecommerce Tracking, Goal tracking, Campaign tracking,
Custom Variables, Email Reports, Custom Segment Editor, Geo Location, Real time maps, and more!
Documentation and more info on http://piwik.org
## Code Status
The Piwik project uses an ever-expanding comprehensive set of thousands of unit tests and dozens of integration [tests](https://github.com/piwik/piwik/tree/master/tests),
running on the hosted distributed continuous integration platform Travis-CI.
Build status (master branch) [![Build Status](https://travis-ci.org/piwik/piwik.png?branch=master)](https://travis-ci.org/piwik/piwik) - Screenshot tests Build [![Build Status](https://travis-ci.org/piwik/piwik-ui-tests.png?branch=master)](https://travis-ci.org/piwik/piwik-ui-tests)

21
www/analytics/SECURITY.md Normal file
View file

@ -0,0 +1,21 @@
# Reporting Security Issues
## Security Bug Bounty Program
The Piwik Security Bug Bounty Program is designed to encourage security research in Piwik software and to reward those who help us create the safest web analytics platform. The bounty for valid critical security bugs is a **$555** (US) cash reward. The bounty for non-critical bugs is **$242** (US), paid via Paypal.
## Responsible disclosure by email
If you have found a security issue in Piwik please read [our security notes](http://piwik.org/security/) regarding responsible disclosures.
[Email your Report Vulnerability to the Piwik Security team](mailto:security@piwik.org?subject=Reporting%20Vulnerability%20in%20Piwik)
## Improve your Piwik Server Security
[Secure Piwik server](http://piwik.org/docs/how-to-secure-piwik/): follow these steps to keep your Piwik data safe.
## Security announcements
Please subscribe to [the Changelog](http://piwik.org/changelog/) ([rss feed](http://piwik.org/changelog/feed/)) to be notified of new releases (including security releases).

41
www/analytics/bower.json Normal file
View file

@ -0,0 +1,41 @@
{
"name": "Piwik",
"main": "piwik.js",
"homepage": "http://piwik.org",
"authors": [
"Piwik.org <hello@piwik.org>"
],
"description": "the leading free/libre analytics platform",
"private": true,
"keywords": [
"piwik",
"web",
"analytics"
],
"dependencies": {
"jquery-ui": "1.10.4",
"jquery": "~1.11.0",
"angular": "~1.2.0",
"angular-sanitize": "~1.2.0",
"angular-animate": "~1.2.0",
"angular-cookies": "~1.2.0",
"angular-mocks": "~1.2.0",
"ngDialog": "~0.2.0",
"html5shiv": "~3.7.0",
"mousetrap": "~1.4.0",
"sprintf": "~1.0.0",
"jScrollPane": "~2.0.0",
"jquery-mousewheel": "~3.1.12",
"jquery-placeholder": "~2.0.8",
"jQuery.dotdotdot": "~1.7.2",
"jquery.scrollTo": "~1.4.13",
"chroma-js": "~0.6.0",
"visibilityjs": "~1.2.1"
},
"license": "GPLv3 or later",
"ignore": [
"**/.*",
"node_modules",
"tests"
]
}

101
www/analytics/composer.json
View file

@ -1,7 +1,7 @@
{
"name": "piwik/piwik",
"type": "application",
"description": "Open Source Real Time Web Analytics Platform",
"description": "the leading free/libre analytics platform",
"keywords": ["piwik","web","analytics"],
"homepage": "http://piwik.org",
"license": "GPL-3.0+",
@ -14,17 +14,98 @@
],
"support": {
"forum": "http://forum.piwik.org/",
"issues": "http://dev.piwik.org/trac/roadmap",
"wiki": "http://dev.piwik.org/",
"issues": "https://github.com/piwik/piwik/issues",
"wiki": "https://github.com/piwik/piwik/wiki",
"source": "https://github.com/piwik/piwik"
},
"autoload": {
"psr-4": {
"Piwik\\Plugins\\": "plugins/",
"Piwik\\": "core/"
},
"psr-0": {
"Zend_": "libs/",
"HTML_": "libs/",
"PEAR_": "libs/",
"Archive_": "libs/"
}
},
"autoload-dev": {
"psr-4": {
"Piwik\\Tests\\": "tests/PHPUnit/"
}
},
"require": {
"php": ">=5.3.2",
"twig/twig": "1.*",
"leafo/lessphp": "~0.3",
"symfony/console": ">=v2.3.5",
"tedivm/jshrink": "v0.5.1",
"mustangostang/spyc": "0.5.*",
"piwik/device-detector": "*"
"php": ">=5.3.3",
"twig/twig": "~1.0",
"leafo/lessphp": "~0.5.0",
"symfony/console": "~2.6.0",
"tedivm/jshrink": "~0.5.1",
"mustangostang/spyc": "~0.5.0",
"piwik/device-detector": "~3.0",
"piwik/decompress": "~1.0",
"piwik/network": "~0.1.0",
"piwik/cache": "~0.2.5",
"piwik/ini": "^1.0.6",
"php-di/php-di": "5.0.0-beta1",
"psr/log": "~1.0",
"monolog/monolog": "~1.11",
"symfony/monolog-bridge": "~2.6.0",
"symfony/event-dispatcher": "~2.6.0",
"pear/pear_exception": "~1.0.0",
"piwik/referrer-spam-blacklist": "~1.0",
"piwik/searchengine-and-social-list": "~1.0",
"tecnickcom/tcpdf": "~6.0",
"piwik/piwik-php-tracker": "^1.0"
},
"require-dev": {
"aws/aws-sdk-php": "2.7.1",
"phpunit/phpunit": "~4.8",
"facebook/xhprof": "dev-master",
"phpseclib/phpseclib": "~0.3.8",
"symfony/var-dumper": "~2.6.0",
"symfony/yaml": "~2.6.0"
},
"repositories": [
{
"type": "package",
"package": {
"name": "facebook/xhprof",
"type": "library",
"description": "XHProf: A Hierarchical Profiler for PHP",
"keywords": ["profiling", "performance"],
"homepage": "http://pecl.php.net/package/xhprof",
"license": "Apache-2.0",
"version": "master",
"require": {
"php": ">=5.2.0"
},
"autoload": {
"files": [
"xhprof_lib/utils/xhprof_lib.php",
"xhprof_lib/utils/xhprof_runs.php"
]
},
"source": {
"type": "git",
"url": "https://github.com/phacility/xhprof",
"reference": "master"
}
}
}
],
"scripts": {
"pre-update-cmd": [
"Piwik\\Composer\\ScriptHandler::cleanXhprof"
],
"pre-install-cmd": [
"Piwik\\Composer\\ScriptHandler::cleanXhprof"
],
"post-update-cmd": [
"Piwik\\Composer\\ScriptHandler::buildXhprof"
],
"post-install-cmd": [
"Piwik\\Composer\\ScriptHandler::buildXhprof"
]
}
}

2541
www/analytics/composer.lock generated
View file

File diff suppressed because it is too large Load diff

17
www/analytics/config/.htaccess
View file

@ -1,13 +1,8 @@
<Files "*">
<IfModule mod_access.c>
Deny from all
</IfModule>
<IfModule !mod_access_compat>
<IfModule mod_authz_host.c>
Deny from all
</IfModule>
</IfModule>
<IfModule mod_access_compat>
Deny from all
</IfModule>
<IfVersion < 2.4>
Deny from all
</IfVersion>
<IfVersion >= 2.4>
Require all denied
</IfVersion>
</Files>

12
www/analytics/config/environment/dev.php Normal file
View file

@ -0,0 +1,12 @@
<?php
return array(
'Piwik\Cache\Backend' => DI\object('Piwik\Cache\Backend\ArrayCache'),
'Piwik\Translation\Loader\LoaderInterface' => DI\object('Piwik\Translation\Loader\LoaderCache')
->constructor(DI\get('Piwik\Translation\Loader\DevelopmentLoader')),
'Piwik\Translation\Loader\DevelopmentLoader' => DI\object()
->constructor(DI\get('Piwik\Translation\Loader\JsonFileLoader')),
);

97
www/analytics/config/environment/test.php Normal file
View file

@ -0,0 +1,97 @@
<?php
use Interop\Container\ContainerInterface;
use Piwik\Common;
use Piwik\Tests\Framework\Mock\FakeAccess;
use Piwik\Tests\Framework\Mock\TestConfig;
return array(
// Disable logging
'Psr\Log\LoggerInterface' => DI\object('Psr\Log\NullLogger'),
'Piwik\Cache\Backend' => function () {
return \Piwik\Cache::buildBackend('file');
},
'cache.eager.cache_id' => 'eagercache-test-',
// Disable loading core translations
'Piwik\Translation\Translator' => DI\decorate(function ($previous, ContainerInterface $c) {
$loadRealTranslations = $c->get('test.vars.loadRealTranslations');
if (!$loadRealTranslations) {
return new \Piwik\Translation\Translator($c->get('Piwik\Translation\Loader\LoaderInterface'), $directories = array());
} else {
return $previous;
}
}),
'Piwik\Config' => DI\decorate(function ($previous, ContainerInterface $c) {
$testingEnvironment = $c->get('Piwik\Tests\Framework\TestingEnvironmentVariables');
$dontUseTestConfig = $c->get('test.vars.dontUseTestConfig');
if (!$dontUseTestConfig) {
$settingsProvider = $c->get('Piwik\Application\Kernel\GlobalSettingsProvider');
return new TestConfig($settingsProvider, $testingEnvironment, $allowSave = false, $doSetTestEnvironment = true);
} else {
return $previous;
}
}),
'Piwik\Access' => DI\decorate(function ($previous, ContainerInterface $c) {
$testUseMockAuth = $c->get('test.vars.testUseMockAuth');
if ($testUseMockAuth) {
$idSitesAdmin = $c->get('test.vars.idSitesAdminAccess');
$access = new FakeAccess();
if (!empty($idSitesAdmin)) {
FakeAccess::$superUser = false;
FakeAccess::$idSitesAdmin = $idSitesAdmin;
FakeAccess::$identity = 'adminUserLogin';
} else {
FakeAccess::$superUser = true;
FakeAccess::$superUserLogin = 'superUserLogin';
}
return $access;
} else {
return $previous;
}
}),
'observers.global' => DI\add(array(
array('AssetManager.getStylesheetFiles', function (&$stylesheets) {
$useOverrideCss = \Piwik\Container\StaticContainer::get('test.vars.useOverrideCss');
if ($useOverrideCss) {
$stylesheets[] = 'tests/resources/screenshot-override/override.css';
}
}),
array('AssetManager.getJavaScriptFiles', function (&$jsFiles) {
$useOverrideJs = \Piwik\Container\StaticContainer::get('test.vars.useOverrideJs');
if ($useOverrideJs) {
$jsFiles[] = 'tests/resources/screenshot-override/override.js';
}
}),
array('Updater.checkForUpdates', function () {
try {
@\Piwik\Filesystem::deleteAllCacheOnUpdate();
} catch (Exception $ex) {
// pass
}
}),
array('Test.Mail.send', function (\Zend_Mail $mail) {
$outputFile = PIWIK_INCLUDE_PATH . '/tmp/' . Common::getRequestVar('module', '') . '.' . Common::getRequestVar('action', '') . '.mail.json';
$outputContent = str_replace("=\n", "", $mail->getBodyText($textOnly = true));
$outputContent = str_replace("=0A", "\n", $outputContent);
$outputContent = str_replace("=3D", "=", $outputContent);
$outputContents = array(
'from' => $mail->getFrom(),
'to' => $mail->getRecipients(),
'subject' => $mail->getSubject(),
'contents' => $outputContent
);
file_put_contents($outputFile, json_encode($outputContents));
}),
)),
);

62
www/analytics/config/environment/ui-test.php Normal file
View file

@ -0,0 +1,62 @@
<?php
use Piwik\Container\StaticContainer;
return array(
// UI tests will remove the port from all URLs to the test server. if a test
// requires the ports in UI tests (eg, Overlay), add the api/controller methods
// to one of these blacklists
'tests.ui.url_normalizer_blacklist.api' => array(),
'tests.ui.url_normalizer_blacklist.controller' => array(),
'Piwik\Config' => \DI\decorate(function (\Piwik\Config $config) {
$config->General['cors_domains'][] = '*';
$config->General['trusted_hosts'][] = $config->tests['http_host'];
$config->General['trusted_hosts'][] = $config->tests['http_host'] . ':' . $config->tests['port'];
return $config;
}),
'observers.global' => \DI\add(array(
// removes port from all URLs to the test Piwik server so UI tests will pass no matter
// what port is used
array('Request.dispatch.end', function (&$result) {
$request = $_GET + $_POST;
$apiblacklist = StaticContainer::get('tests.ui.url_normalizer_blacklist.api');
if (!empty($request['method'])
&& in_array($request['method'], $apiblacklist)
) {
return;
}
$controllerActionblacklist = StaticContainer::get('tests.ui.url_normalizer_blacklist.controller');
if (!empty($request['module'])
&& !empty($request['action'])
) {
$controllerAction = $request['module'] . '.' . $request['action'];
if (in_array($controllerAction, $controllerActionblacklist)) {
return;
}
}
$config = \Piwik\Config::getInstance();
$host = $config->tests['http_host'];
$port = $config->tests['port'];
if (!empty($port)) {
// remove the port from URLs if any so UI tests won't fail if the port isn't 80
$result = str_replace($host . ':' . $port, $host, $result);
}
// remove PIWIK_INCLUDE_PATH from result so tests don't change based on the machine used
$result = str_replace(realpath(PIWIK_INCLUDE_PATH), '', $result);
}),
array('Controller.ExampleRssWidget.rssPiwik.end', function (&$result, $parameters) {
$result = "";
}),
)),
);

322
www/analytics/config/global.ini.php
View file

@ -18,25 +18,46 @@ password =
dbname =
tables_prefix =
port = 3306
adapter = PDO_MYSQL
adapter = PDO\MYSQL
type = InnoDB
schema = Mysql
; if charset is set to utf8, Piwik will ensure that it is storing its data using UTF8 charset.
; it will add a sql query SET at each page view.
; Piwik should work correctly without this setting.
;charset = utf8
; Piwik should work correctly without this setting but we recommend to have a charset set.
charset = utf8
[database_tests]
host = localhost
username = root
username = "@USERNAME@"
password =
dbname = piwik_tests
tables_prefix = piwiktests_
port = 3306
adapter = PDO_MYSQL
adapter = PDO\MYSQL
type = InnoDB
schema = Mysql
charset = utf8
[tests]
; needed in order to run tests.
; if Piwik is available at http://localhost/dev/piwik/ replace @REQUEST_URI@ with /dev/piwik/
; note: the REQUEST_URI should not contain "plugins" or "tests" in the PATH
http_host = localhost
remote_addr = "127.0.0.1"
request_uri = "@REQUEST_URI@"
port =
; access key and secret as listed in AWS -> IAM -> Users
aws_accesskey = ""
aws_secret = ""
; key pair name as listed in AWS -> EC2 -> Key Pairs. Key name should be different per user.
aws_keyname = ""
; PEM file can be downloaded after creating a new key pair in AWS -> EC2 -> Key Pairs
aws_pem_file = "<path to pem file>"
aws_securitygroups[] = "default"
aws_region = "us-east-1"
aws_ami = "ami-ac24bac4"
aws_instance_type = "c3.large"
[log]
; possible values for log: screen, database, file
@ -44,20 +65,36 @@ log_writers[] = screen
; log level, everything logged w/ this level or one of greater severity
; will be logged. everything else will be ignored. possible values are:
; NONE, ERROR, WARN, INFO, DEBUG, VERBOSE
; ERROR, WARN, INFO, DEBUG
log_level = WARN
; if set to 1, only requests done in CLI mode (eg. the archive.php cron run) will be logged
; NOTE: log_only_when_debug_parameter will also be checked for
log_only_when_cli = 0
; if set to 1, only requests with "&debug" parameter will be logged
; NOTE: log_only_when_cli will also be checked for
log_only_when_debug_parameter = 0
; if configured to log in a file, log entries will be made to this file
logger_file_path = tmp/logs/piwik.log
[Cache]
; available backends are 'file', 'array', 'null', 'redis', 'chained'
; 'array' will cache data only during one request
; 'null' will not cache anything at all
; 'file' will cache on the filesystem
; 'redis' will cache on a Redis server, use this if you are running Piwik with multiple servers. Further configuration in [RedisCache] is needed
; 'chained' will chain multiple cache backends. Further configuration in [ChainedCache] is needed
backend = chained
[ChainedCache]
; The chained cache will always try to read from the fastest backend first (the first listed one) to avoid requesting
; the same cache entry from the slowest backend multiple times in one request.
backends[] = array
backends[] = file
[RedisCache]
; Redis server configuration.
host = "127.0.0.1"
port = 6379
timeout = 0.0
password = ""
database = 14
; In case you are using queued tracking: Make sure to configure a different database! Otherwise queued requests might
; be flushed
[Debug]
; if set to 1, the archiving process will always be triggered, even if the archive has already been computed
@ -72,40 +109,68 @@ always_archive_data_range = 0;
; NOTE: you must also set [log] log_writers[] = "screen" to enable the profiler to print on screen
enable_sql_profiler = 0
; if set to 1, a Piwik tracking code will be included in the Piwik UI footer and will track visits, pages, etc. to idsite = 1
; this is useful for Piwik developers as an easy way to create data in their local Piwik
track_visits_inside_piwik_ui = 0
; if set to 1, javascript files will be included individually and neither merged nor minified.
; this option must be set to 1 when adding, removing or modifying javascript files
disable_merged_assets = 0
; If set to 1, all requests to piwik.php will be forced to be 'new visitors'
tracker_always_new_visitor = 0
; Allow automatic upgrades to Beta or RC releases
allow_upgrades_to_beta = 0
; if set to 1, all SQL queries will be logged using the DEBUG log level
log_sql_queries = 0
[DebugTests]
; Set to 1 by default. If you set to 0, the standalone plugins (with their own git repositories)
; will not be loaded when executing tests.
enable_load_standalone_plugins_during_tests = 1
; When set to 1, standalone plugins (those with their own git repositories)
; will be loaded when executing tests.
enable_load_standalone_plugins_during_tests = 0
[Development]
; Enables the development mode where we avoid most caching to make sure code changes will be directly applied as
; some caches are only invalidated after an update otherwise. When enabled it'll also performs some validation checks.
; For instance if you register a method in a widget we will verify whether the method actually exists and is public.
; If not, we will show you a helpful warning to make it easy to find simple typos etc.
enabled = 0
; if set to 1, javascript files will be included individually and neither merged nor minified.
; this option must be set to 1 when adding, removing or modifying javascript files
; Note that for quick debugging, instead of using below setting, you can add `&disable_merged_assets=1` to the Piwik URL
disable_merged_assets = 0
[General]
; the following settings control whether Unique Visitors will be processed for different period types.
; the following settings control whether Unique Visitors `nb_uniq_visitors` and Unique users `nb_users` will be processed for different period types.
; year and range periods are disabled by default, to ensure optimal performance for high traffic Piwik instances
; if you set it to 1 and want the Unique Visitors to be re-processed for reports in the past, drop all piwik_archive_* tables
; it is recommended to always enable Unique Visitors processing for 'day' periods
; it is recommended to always enable Unique Visitors and Unique Users processing for 'day' periods
enable_processing_unique_visitors_day = 1
enable_processing_unique_visitors_week = 1
enable_processing_unique_visitors_month = 1
enable_processing_unique_visitors_year = 0
enable_processing_unique_visitors_range = 0
; controls whether Unique Visitors will be processed for groups of websites. these metrics describe the number
; of unique visitors across the entire set of websites, so if a visitor visited two websites in the group, she
; would still only be counted as one. only relevant when using plugins that group sites together
enable_processing_unique_visitors_multiple_sites = 0
; The list of periods that are available in the Piwik calendar
; Example use case: custom date range requests are processed in real time,
; so they may take a few minutes on very high traffic website: you may remove "range" below to disable this period
enabled_periods_UI = "day,week,month,year,range"
enabled_periods_API = "day,week,month,year,range"
; whether to enable subquery cache for Custom Segment archiving queries
enable_segments_subquery_cache = 0
; Any segment subquery that matches more than segments_subquery_cache_limit IDs will not be cached,
; and the original subquery executed instead.
segments_subquery_cache_limit = 100000
; TTL: Time to live for cache files, in seconds. Default to 60 minutes
segments_subquery_cache_ttl = 3600
; when set to 1, all requests to Piwik will return a maintenance message without connecting to the DB
; this is useful when upgrading using the shell command, to prevent other users from accessing the UI while Upgrade is in progress
maintenance_mode = 0
; Defines the release channel that shall be used. Currently available values are:
; "latest_stable", "latest_beta", "latest_2x_stable", "latest_2x_beta"
release_channel = "latest_stable"
; character used to automatically create categories in the Actions > Pages, Outlinks and Downloads reports
; for example a URL like "example.com/blog/development/first-post" will create
; the page first-post in the subcategory development which belongs to the blog category
@ -143,10 +208,36 @@ browser_archiving_disabled_enforce = 0
; By default, users can create Segments which are to be processed in Real-time.
; Setting this to 0 will force all newly created Custom Segments to be "Pre-processed (faster, requires archive.php cron)"
; This can be useful if you want to prevent users from adding much load on the server.
; Note: any existing Segment set to "processed in Real time", will still be set to Real-time.
; this will only affect custom segments added or modified after this setting is changed.
; Notes:
; * any existing Segment set to "processed in Real time", will still be set to Real-time.
; this will only affect custom segments added or modified after this setting is changed.
; * when set to 0 then any user with at least 'view' access will be able to create pre-processed segments.
enable_create_realtime_segments = 1
; Whether to enable the "Suggest values for segment" in the Segment Editor panel.
; Set this to 0 in case your Piwik database is very big, and suggested values may not appear in time
enable_segment_suggested_values = 1
; By default, any user with a "view" access for a website can create segment assigned to this website.
; Set this to "admin" or "superuser" to require that users should have at least this access to create new segments.
; Note: anonymous user (even if it has view access) is not allowed to create or edit segment.
; Possible values are "view", "admin", "superuser"
adding_segment_requires_access = "view"
; Whether it is allowed for users to add segments that affect all websites or not. If there are many websites
; this admin option can be used to prevent users from performing an action that will have a major impact
; on Piwik performance.
allow_adding_segments_for_all_websites = 1
; When archiving segments for the first time, this determines the oldest date that will be archived.
; This option can be used to avoid archiving (for isntance) the lastN years for every new segment.
; Valid option values include: "beginning_of_time" (start date of archiving will not be changed)
; "segment_last_edit_time" (start date of archiving will be the earliest last edit date found,
; if none is found, the created date is used)
; "segment_creation_time" (start date of archiving will be the creation date of the segment)
; lastN where N is an integer (eg "last10" to archive for 10 days before the segment creation date)
process_new_segments_from = "beginning_of_time"
; this action name is used when the URL ends with a slash /
; it is useful to have an actual string to write in the UI
action_default_name = index
@ -158,6 +249,11 @@ default_language = en
; default number of elements in the datatable
datatable_default_limit = 10
; Each datatable report has a Row Limit selector at the bottom right.
; By default you can select from 5 to 500 rows. You may customise the values below
; -1 will be displayed as 'all' and it will export all rows (filter_limit=-1)
datatable_row_limits = "5,10,25,50,100,250,500,-1"
; default number of rows returned in API responses
; this value is overwritten by the '# Rows to display' selector.
; if set to -1, a click on 'Export as' will export all rows independently of the current '# Rows to display'.
@ -175,21 +271,32 @@ default_day = yesterday
default_period = day
; Time in seconds after which an archive will be computed again. This setting is used only for today's statistics.
; Defaults to 10 seconds so that by default, Piwik provides real time reporting.
; This setting is overriden in the UI, under "General Settings".
; This setting is only used if it hasn't been overriden via the UI yet, or if enable_general_settings_admin=0
time_before_today_archive_considered_outdated = 10
time_before_today_archive_considered_outdated = 150
; This setting is overriden in the UI, under "General Settings".
; The default value is to allow browsers to trigger the Piwik archiving process.
; This setting is only used if it hasn't been overriden via the UI yet, or if enable_general_settings_admin=0
enable_browser_archiving_triggering = 1
; By default, Piwik will force archiving of range periods from browser requests, even if enable_browser_archiving_triggering
; is set to 0. This can sometimes create too much of a demand on system resources. Setting this option to 0 and setting
; enable_browser_archiving_triggering to 0 will make sure ranges are not archived on browser request. Since the cron
; archiver does not archive ranges, you must either disable ranges or make sure the ranges users' want to see will be
; processed somehow.
archiving_range_force_on_browser_request = 1
; By default Piwik runs OPTIMIZE TABLE SQL queries to free spaces after deleting some data.
; If your Piwik tracks millions of pages, the OPTIMIZE TABLE queries might run for hours (seen in "SHOW FULL PROCESSLIST \g")
; so you can disable these special queries here:
enable_sql_optimize_queries = 1
; By default Piwik is purging complete date range archives to free spaces after deleting some data.
; If you are pre-processing custom ranges using CLI task to make them easily available in UI,
; you can prevent this action from happening by setting this parameter to value bigger than 1
purge_date_range_archives_after_X_days = 1
; MySQL minimum required version
; note: timezone support added in 4.1.3
minimum_mysql_version = 4.1
@ -200,7 +307,7 @@ minimum_pgsql_version = 8.3
; Minimum adviced memory limit in php.ini file (see memory_limit value)
minimum_memory_limit = 128
; Minimum memory limit enforced when archived via misc/cron/archive.php
; Minimum memory limit enforced when archived via ./console core:archive
minimum_memory_limit_when_archiving = 768
; Piwik will check that usernames and password have a minimum length, and will check that characters are "allowed"
@ -217,14 +324,15 @@ session_save_handler = files
; If set to 1, Piwik will automatically redirect all http:// requests to https://
; If SSL / https is not correctly configured on the server, this will break Piwik
; If you set this to 1, and your SSL configuration breaks later on, you can always edit this back to 0
; If you set this to 1, and your SSL configuration breaks later on, you can always edit this back to 0
; it is recommended for security reasons to always use Piwik over https
force_ssl = 0
; login cookie name
login_cookie_name = piwik_auth
; login cookie expiration (14 days)
; By default, the auth cookie is set only for the duration of session.
; if "Remember me" is checked, the auth cookie will be valid for 14 days by default
login_cookie_expire = 1209600
; The path on the server in which the cookie will be available on.
@ -237,6 +345,12 @@ login_password_recovery_email_address = "password-recovery@{DOMAIN}"
; name that appears as a Sender in the password recovery email
login_password_recovery_email_name = Piwik
; email address that appears as a Repy-to in the password recovery email
; if specified, {DOMAIN} will be replaced by the current Piwik domain
login_password_recovery_replyto_email_address = "no-reply@{DOMAIN}"
; name that appears as a Reply-to in the password recovery email
login_password_recovery_replyto_email_name = "No-reply"
; By default when user logs out he is redirected to Piwik "homepage" usually the Login form.
; Uncomment the next line to set a URL to redirect the user to after he logs out of Piwik.
; login_logout_url = http://...
@ -255,10 +369,21 @@ language_cookie_name = piwik_lang
; standard email address displayed when sending emails
noreply_email_address = "noreply@{DOMAIN}"
; standard email name displayed when sending emails. If not set, a default name will be used.
noreply_email_name = ""
; feedback email address;
; when testing, use your own email address or "nobody"
feedback_email_address = "feedback@piwik.org"
; using to set reply_to in reports e-mail to login of report creator
scheduled_reports_replyto_is_user_email_and_alias = 0
; scheduled reports truncate limit
; the report will be rendered with the first 23 rows and will aggregate other rows in a summary row
; 23 rows table fits in one portrait page
scheduled_reports_truncate = 23
; during archiving, Piwik will limit the number of results recorded, for performance reasons
; maximum number of rows for any of the Referrers tables (keywords, search engines, campaigns, etc.)
datatable_archiving_maximum_rows_referrers = 1000
@ -347,9 +472,20 @@ enable_trusted_host_check = 1
;trusted_hosts[] = example.com
;trusted_hosts[] = stats.example.com
; The release server is an essential part of the Piwik infrastructure/ecosystem
; to provide the latest software version.
latest_version_url = http://builds.piwik.org/latest.zip
; List of Cross-origin resource sharing domains (eg domain or subdomain names) when generating absolute URLs.
; Described here: http://en.wikipedia.org/wiki/Cross-origin_resource_sharing
;
; Examples:
;cors_domains[] = http://example.com
;cors_domains[] = http://stats.example.com
;
; Or you may allow cross domain requests for all domains with:
;cors_domains[] = *
; If you use this Piwik instance over multiple hostnames, Piwik will need to know
; a unique instance_id for this instance, so that Piwik can serve the right custom logo and tmp/* assets,
; independently of the hostname Piwik is currently running under.
; instance_id = stats.example.com
; The API server is an essential part of the Piwik infrastructure/ecosystem to
; provide services to Piwik installations, e.g., getLatestVersion and
@ -362,6 +498,10 @@ api_service_url = http://api.piwik.org
; eg. $period=range&date=previous10 becomes $period=day&date=previous10. Use this setting to override the $period value.
graphs_default_period_to_plot_when_period_range = day
; When the ImageGraph plugin is activated, enabling this option causes the image graphs to show the evolution
; within the selected period instead of the evolution across the last n periods.
graphs_show_evolution_within_selected_period = 0
; The Overlay plugin shows the Top X following pages, Top X downloads and Top X outlinks which followed
; a view of the current page. The value X can be set here.
overlay_following_pages_limit = 300
@ -411,7 +551,29 @@ enable_auto_update = 1
; If set to 0 it also disables the "sent plugin update emails" feature in general and the related setting in the UI.
enable_update_communication = 1
; This controls whether the pivotBy query parameter can be used with any dimension or just subtable
; dimensions. If set to 1, it will fetch a report with a segment for each row of the table being pivoted.
; At present, this is very inefficient, so it is disabled by default.
pivot_by_filter_enable_fetch_by_segment = 0
; This controls the default maximum number of columns to display in a pivot table. Since a pivot table displays
; a table's rows as columns, the number of columns can become very large, which will affect webpage layouts.
; Set to -1 to specify no limit. Note: The pivotByColumnLimit query parameter can be used to override this default
; on a per-request basis;
pivot_by_filter_default_column_limit = 10
; If set to 0 it will disable Piwik Pro advertisements in some places. For example in the installation screen, the
; Piwik Pro Ad widget will be removed etc.
piwik_pro_ads_enabled = 1
[Tracker]
; Piwik uses "Privacy by default" model. When one of your users visit multiple of your websites tracked in this Piwik,
; Piwik will create for this user a fingerprint that will be different across the multiple websites.
; If you want to track unique users across websites (for example when using the InterSites plugin) you may set this setting to 1.
; Note: setting this to 0 increases your users' privacy.
enable_fingerprinting_across_websites = 0
; Piwik uses first party cookies by default. If set to 1,
; the visit ID cookie will be set on the Piwik server domain as well
; this is useful when you want to do cross websites analysis
@ -421,14 +583,14 @@ use_third_party_id_cookie = 0
; Once enabled (set to 1) messages will be logged to all loggers defined in "[log] log_writers" config.
debug = 0
; There is a feature in the Tracking API that lets you create new visit at any given time, for example if you know that a different user/customer is using
; the app then you would want to tell Piwik to create a new visit (even though both users are using the same browser/computer).
; To prevent abuse and easy creation of fake visits, this feature requires admin token_auth by default
; If you wish to use this feature using the Javascript tracker, you can set the setting new_visit_api_requires_admin=0, and in Javascript write:
; _paq.push(['appendToTrackingUrl', 'new_visit=1']);
new_visit_api_requires_admin = 1
; This option is an alternative to the debug option above. When set to 1, you can debug tracker request by adding
; a debug=1 query paramater in the URL. All other HTTP requests will not have debug enabled. For security reasons this
; option should be only enabled if really needed and only for a short time frame. Otherwise anyone can set debug=1 and
; see the log output as well.
debug_on_demand = 0
; This setting should only be set to 1 in an intranet setting, where most users have the same configuration (browsers, OS)
; This setting is described in this FAQ: http://piwik.org/faq/how-to/faq_175/
; Note: generally this should only be set to 1 in an intranet setting, where most users have the same configuration (browsers, OS)
; and the same IP. If left to 0 in this setting, all visitors will be counted as one single visitor.
trust_visitors_cookies = 0
@ -436,9 +598,9 @@ trust_visitors_cookies = 0
; This is used only if use_third_party_id_cookie = 1
cookie_name = _pk_uid
; by default, the Piwik tracking cookie expires in 2 years
; by default, the Piwik tracking cookie expires in 13 months (365 + 28 days)
; This is used only if use_third_party_id_cookie = 1
cookie_expire = 63072000
cookie_expire = 33955200;
; The path on the server in which the cookie will be available on.
; Defaults to empty. See spec in http://curl.haxx.se/rfc/cookie_spec.html
@ -466,7 +628,7 @@ default_time_one_page_visit = 0
; The mapping is defined in core/DataFiles/LanguageToCountry.php,
enable_language_to_country_guess = 1
; When the misc/cron/archive.php cron hasn't been setup, we still need to regularly run some maintenance tasks.
; When the `./console core:archive` cron hasn't been setup, we still need to regularly run some maintenance tasks.
; Visits to the Tracker will try to trigger Scheduled Tasks (eg. scheduled PDF/HTML reports by email).
; Scheduled tasks will only run if 'Enable Piwik Archiving from Browser' is enabled in the General Settings.
; Tasks run once every hour maximum, they might not run every hour if traffic is low.
@ -479,13 +641,27 @@ ignore_visits_cookie_name = piwik_ignore
; Comma separated list of variable names that will be read to define a Campaign name, for example CPC campaign
; Example: If a visitor first visits 'index.php?piwik_campaign=Adwords-CPC' then it will be counted as a campaign referrer named 'Adwords-CPC'
; Includes by default the GA style campaign parameters
campaign_var_name = "pk_campaign,piwik_campaign,utm_campaign,utm_source,utm_medium"
campaign_var_name = "pk_cpn,pk_campaign,piwik_campaign,utm_campaign,utm_source,utm_medium"
; Comma separated list of variable names that will be read to track a Campaign Keyword
; Example: If a visitor first visits 'index.php?piwik_campaign=Adwords-CPC&piwik_kwd=My killer keyword' ;
; then it will be counted as a campaign referrer named 'Adwords-CPC' with the keyword 'My killer keyword'
; Includes by default the GA style campaign keyword parameter utm_term
campaign_keyword_var_name = "pk_kwd,piwik_kwd,pk_keyword,utm_term"
campaign_keyword_var_name = "pk_kwd,pk_keyword,piwik_kwd,utm_term"
; if set to 1, actions that contain different campaign information from the visitor's ongoing visit will
; be treated as the start of a new visit. This will include situations when campaign information was absent before,
; but is present now.
create_new_visit_when_campaign_changes = 1
; if set to 1, actions that contain different website referrer information from the visitor's ongoing visit
; will be treated as the start of a new visit. This will include situations when website referrer information was
; absent before, but is present now.
create_new_visit_when_website_referrer_changes = 0
; ONLY CHANGE THIS VALUE WHEN YOU DO NOT USE PIWIK ARCHIVING, SINCE THIS COULD CAUSE PARTIALLY MISSING ARCHIVE DATA
; Whether to force a new visit at midnight for every visitor. Default 1.
create_new_visit_after_midnight = 1
; maximum length of a Page Title or a Page URL recorded in the log_action.name table
page_maximum_length = 1024;
@ -494,9 +670,16 @@ page_maximum_length = 1024;
; TTL: Time to live for cache files, in seconds. Default to 5 minutes.
tracker_cache_file_ttl = 300
; Whether Bulk tracking requests to the Tracking API requires the token_auth to be set.
bulk_requests_require_authentication = 0
; Whether Bulk tracking requests will be wrapped within a DB Transaction.
; This greatly increases performance of Log Analytics and in general any Bulk Tracking API requests.
bulk_requests_use_transaction = 1
; DO NOT USE THIS SETTING ON PUBLICLY AVAILABLE PIWIK SERVER
; !!! Security risk: if set to 0, it would allow anyone to push data to Piwik with custom dates in the past/future and even with fake IPs!
; When using the Tracking API, to override either the datetime and/or the visitor IP,
; When using the Tracking API, to override either the datetime and/or the visitor IP,
; token_auth with an "admin" access is required. If you set this setting to 0, the token_auth will not be required anymore.
; DO NOT USE THIS SETTING ON PUBLIC PIWIK SERVERS
tracking_requests_require_authentication = 1
@ -509,7 +692,7 @@ tracking_requests_require_authentication = 1
; for which all reports should be Archived during the cron execution
; All segment values MUST be URL encoded.
;Segments[]="visitorType==new"
;Segments[]="visitorType==returning"
;Segments[]="visitorType==returning,visitorType==returningCustomer"
; If you define Custom Variables for your visitor, for example set the visit type
;Segments[]="customVariableName1==VisitType;customVariableValue1==Customer"
@ -554,9 +737,12 @@ username = ; Proxy username: optional; if specified, password is mandatory
password = ; Proxy password: optional; if specified, username is mandatory
[Plugins]
; list of plugins (in order they will be loaded) that are activated by default in the Piwik platform
Plugins[] = CorePluginsAdmin
Plugins[] = CoreAdminHome
Plugins[] = CoreHome
Plugins[] = WebsiteMeasurable
Plugins[] = Diagnostics
Plugins[] = CoreVisualizations
Plugins[] = Proxy
Plugins[] = API
@ -568,8 +754,10 @@ Plugins[] = Actions
Plugins[] = Dashboard
Plugins[] = MultiSites
Plugins[] = Referrers
Plugins[] = UserSettings
Plugins[] = UserLanguage
Plugins[] = DevicesDetection
Plugins[] = Goals
Plugins[] = Ecommerce
Plugins[] = SEO
Plugins[] = Events
Plugins[] = UserCountry
@ -579,8 +767,8 @@ Plugins[] = VisitTime
Plugins[] = VisitorInterest
Plugins[] = ExampleAPI
Plugins[] = ExampleRssWidget
Plugins[] = Provider
Plugins[] = Feedback
Plugins[] = Monolog
Plugins[] = Login
Plugins[] = UsersManager
@ -599,29 +787,31 @@ Plugins[] = MobileMessaging
Plugins[] = Overlay
Plugins[] = SegmentEditor
Plugins[] = Insights
Plugins[] = Morpheus
Plugins[] = Contents
Plugins[] = BulkTracking
Plugins[] = Resolution
Plugins[] = DevicePlugins
Plugins[] = Heartbeat
Plugins[] = Intl
Plugins[] = PiwikPro
[PluginsInstalled]
PluginsInstalled[] = Diagnostics
PluginsInstalled[] = Login
PluginsInstalled[] = CoreAdminHome
PluginsInstalled[] = UsersManager
PluginsInstalled[] = SitesManager
PluginsInstalled[] = Installation
[Plugins_Tracker]
Plugins_Tracker[] = Provider
Plugins_Tracker[] = Goals
Plugins_Tracker[] = PrivacyManager
Plugins_Tracker[] = UserCountry
Plugins_Tracker[] = Login
PluginsInstalled[] = Monolog
PluginsInstalled[] = Intl
[APISettings]
; Any key/value pair can be added in this section, they will be available via the REST call
; index.php?module=API&method=API.getSettings
; index.php?module=API&method=API.getSettings
; This can be used to expose values from Piwik, to control for example a Mobile app tracking
SDK_batch_size = 10
SDK_interval_value = 30
; NOTE: do not directly edit this file! See notice at the top

85
www/analytics/config/global.php Normal file
View file

@ -0,0 +1,85 @@
<?php
use Interop\Container\ContainerInterface;
use Interop\Container\Exception\NotFoundException;
use Piwik\Cache\Eager;
use Piwik\SettingsServer;
return array(
'path.root' => PIWIK_USER_PATH,
'path.tmp' => function (ContainerInterface $c) {
$root = $c->get('path.root');
// TODO remove that special case and instead have plugins override 'path.tmp' to add the instance id
if ($c->has('ini.General.instance_id')) {
$instanceId = $c->get('ini.General.instance_id');
$instanceId = $instanceId ? '/' . $instanceId : '';
} else {
$instanceId = '';
}
return $root . '/tmp' . $instanceId;
},
'path.cache' => DI\string('{path.tmp}/cache/tracker/'),
'Piwik\Cache\Eager' => function (ContainerInterface $c) {
$backend = $c->get('Piwik\Cache\Backend');
$cacheId = $c->get('cache.eager.cache_id');
if (SettingsServer::isTrackerApiRequest()) {
$eventToPersist = 'Tracker.end';
$cacheId .= 'tracker';
} else {
$eventToPersist = 'Request.dispatch.end';
$cacheId .= 'ui';
}
$cache = new Eager($backend, $cacheId);
\Piwik\Piwik::addAction($eventToPersist, function () use ($cache) {
$cache->persistCacheIfNeeded(43200);
});
return $cache;
},
'Piwik\Cache\Backend' => function (ContainerInterface $c) {
try {
$backend = $c->get('ini.Cache.backend');
} catch (NotFoundException $ex) {
$backend = 'chained'; // happens if global.ini.php is not available
}
return \Piwik\Cache::buildBackend($backend);
},
'cache.eager.cache_id' => function () {
return 'eagercache-' . str_replace(array('.', '-'), '', \Piwik\Version::VERSION) . '-';
},
'Psr\Log\LoggerInterface' => DI\object('Psr\Log\NullLogger'),
'Piwik\Translation\Loader\LoaderInterface' => DI\object('Piwik\Translation\Loader\LoaderCache')
->constructor(DI\get('Piwik\Translation\Loader\JsonFileLoader')),
'observers.global' => array(),
'Piwik\EventDispatcher' => DI\object()->constructorParameter('observers', DI\get('observers.global')),
'Zend_Validate_EmailAddress' => function () {
return new \Zend_Validate_EmailAddress(array(
'hostname' => new \Zend_Validate_Hostname(array(
'tld' => false,
))));
},
'Piwik\Tracker\VisitorRecognizer' => DI\object()
->constructorParameter('trustCookiesOnly', DI\get('ini.Tracker.trust_visitors_cookies'))
->constructorParameter('visitStandardLength', DI\get('ini.Tracker.visit_standard_length'))
->constructorParameter('lookbackNSecondsCustom', DI\get('ini.Tracker.window_look_back_for_visitor'))
->constructorParameter('trackerAlwaysNewVisitor', DI\get('ini.Debug.tracker_always_new_visitor')),
'Piwik\Tracker\Settings' => DI\object()
->constructorParameter('isSameFingerprintsAcrossWebsites', DI\get('ini.Tracker.enable_fingerprinting_across_websites')),
);

7268
www/analytics/config/manifest.inc.php
View file

File diff suppressed because it is too large Load diff

25
www/analytics/console
View file

@ -3,26 +3,25 @@
if (!defined('PIWIK_DOCUMENT_ROOT')) {
define('PIWIK_DOCUMENT_ROOT', dirname(__FILE__) == '/' ? '' : dirname(__FILE__));
}
if (file_exists(PIWIK_DOCUMENT_ROOT . '/bootstrap.php')) {
require_once PIWIK_DOCUMENT_ROOT . '/bootstrap.php';
}
if (!defined('PIWIK_INCLUDE_PATH')) {
define('PIWIK_INCLUDE_PATH', PIWIK_DOCUMENT_ROOT);
}
if (!defined('PIWIK_USER_PATH')) {
define('PIWIK_USER_PATH', PIWIK_DOCUMENT_ROOT);
}
require_once PIWIK_INCLUDE_PATH . '/core/testMinimumPhpVersion.php';
require_once file_exists(PIWIK_INCLUDE_PATH . '/vendor/autoload.php')
? PIWIK_INCLUDE_PATH . '/vendor/autoload.php' // Piwik is the main project
: PIWIK_INCLUDE_PATH . '/../../autoload.php'; // Piwik is installed as a dependency
require_once PIWIK_INCLUDE_PATH . '/core/Loader.php';
require_once PIWIK_INCLUDE_PATH . '/libs/upgradephp/upgrade.php';
Piwik\Translate::loadEnglishTranslation();
require_once PIWIK_INCLUDE_PATH . '/core/bootstrap.php';
if (!Piwik\Common::isPhpCliMode()) {
exit;
}
if (!defined('PIWIK_ENABLE_ERROR_HANDLER') || PIWIK_ENABLE_ERROR_HANDLER) {
Piwik\ErrorHandler::registerErrorHandler();
Piwik\ExceptionHandler::setUp();
}
$console = new Piwik\Console();
$console->init();
$console->run();
$console->run();

17
www/analytics/core/.htaccess
View file

@ -1,13 +1,8 @@
<Files "*">
<IfModule mod_access.c>
Deny from all
</IfModule>
<IfModule !mod_access_compat>
<IfModule mod_authz_host.c>
Deny from all
</IfModule>
</IfModule>
<IfModule mod_access_compat>
Deny from all
</IfModule>
<IfVersion < 2.4>
Deny from all
</IfVersion>
<IfVersion >= 2.4>
Require all denied
</IfVersion>
</Files>

131
www/analytics/core/API/ApiRenderer.php Normal file
View file

@ -0,0 +1,131 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable\Renderer;
use Piwik\DataTable;
use Piwik\Piwik;
use Piwik\Plugin;
/**
* API renderer
*/
abstract class ApiRenderer
{
protected $request;
final public function __construct($request)
{
$this->request = $request;
$this->init();
}
protected function init()
{
}
abstract public function sendHeader();
public function renderSuccess($message)
{
return 'Success:' . $message;
}
public function renderException($message, \Exception $exception)
{
return $message;
}
public function renderScalar($scalar)
{
$dataTable = new DataTable\Simple();
$dataTable->addRowsFromArray(array($scalar));
return $this->renderDataTable($dataTable);
}
public function renderDataTable($dataTable)
{
$renderer = $this->buildDataTableRenderer($dataTable);
return $renderer->render();
}
public function renderArray($array)
{
$renderer = $this->buildDataTableRenderer($array);
return $renderer->render();
}
public function renderObject($object)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
public function renderResource($resource)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
/**
* @param $dataTable
* @return Renderer
*/
protected function buildDataTableRenderer($dataTable)
{
$format = self::getFormatFromClass(get_class($this));
if ($format == 'json2') {
$format = 'json';
}
$renderer = Renderer::factory($format);
$renderer->setTable($dataTable);
$renderer->setRenderSubTables(Common::getRequestVar('expanded', false, 'int', $this->request));
$renderer->setHideIdSubDatableFromResponse(Common::getRequestVar('hideIdSubDatable', false, 'int', $this->request));
return $renderer;
}
/**
* @param string $format
* @param array $request
* @return ApiRenderer
* @throws Exception
*/
public static function factory($format, $request)
{
$formatToCheck = '\\' . ucfirst(strtolower($format));
$rendererClassnames = Plugin\Manager::getInstance()->findMultipleComponents('Renderer', 'Piwik\\API\\ApiRenderer');
foreach ($rendererClassnames as $klassName) {
if (Common::stringEndsWith($klassName, $formatToCheck)) {
return new $klassName($request);
}
}
$availableRenderers = array();
foreach ($rendererClassnames as $rendererClassname) {
$availableRenderers[] = self::getFormatFromClass($rendererClassname);
}
$availableRenderers = implode(', ', $availableRenderers);
Common::sendHeader('Content-Type: text/plain; charset=utf-8');
throw new Exception(Piwik::translate('General_ExceptionInvalidRendererFormat', array($format, $availableRenderers)));
}
private static function getFormatFromClass($klassname)
{
$klass = explode('\\', $klassname);
return strtolower(end($klass));
}
}

41
www/analytics/core/API/CORSHandler.php Normal file
View file

@ -0,0 +1,41 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Piwik\Url;
class CORSHandler
{
/**
* @var array
*/
protected $domains;
public function __construct()
{
$this->domains = Url::getCorsHostsFromConfig();
}
public function handle()
{
// allow Piwik to serve data to all domains
if (in_array("*", $this->domains)) {
header('Access-Control-Allow-Origin: *');
return;
}
// specifically allow if it is one of the whitelisted CORS domains
if (!empty($_SERVER['HTTP_ORIGIN'])) {
$origin = $_SERVER['HTTP_ORIGIN'];
if (in_array($origin, $this->domains, true)) {
header('Access-Control-Allow-Origin: ' . $_SERVER['HTTP_ORIGIN']);
}
}
}
}

174
www/analytics/core/API/DataTableGenericFilter.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -11,20 +11,37 @@ namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\Filter\AddColumnsProcessedMetricsGoal;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
class DataTableGenericFilter
{
private static $genericFiltersInfo = null;
/**
* List of filter names not to run.
*
* @var string[]
*/
private $disabledFilters = array();
/**
* @var Report
*/
private $report;
/**
* @var array
*/
private $request;
/**
* Constructor
*
* @param $request
*/
function __construct($request)
public function __construct($request, $report)
{
$this->request = $request;
$this->report = $report;
}
/**
@ -37,6 +54,16 @@ class DataTableGenericFilter
$this->applyGenericFilters($table);
}
/**
* Makes sure a set of filters are not run.
*
* @param string[] $filterNames The name of each filter to disable.
*/
public function disableFilters($filterNames)
{
$this->disabledFilters = array_unique(array_merge($this->disabledFilters, $filterNames));
}
/**
* Returns an array containing the information of the generic Filter
* to be applied automatically to the data resulting from the API calls.
@ -51,43 +78,54 @@ class DataTableGenericFilter
*/
public static function getGenericFiltersInformation()
{
if (is_null(self::$genericFiltersInfo)) {
self::$genericFiltersInfo = array(
'Pattern' => array(
'filter_column' => array('string', 'label'),
'filter_pattern' => array('string'),
),
'PatternRecursive' => array(
'filter_column_recursive' => array('string', 'label'),
'filter_pattern_recursive' => array('string'),
),
'ExcludeLowPopulation' => array(
'filter_excludelowpop' => array('string'),
'filter_excludelowpop_value' => array('float', '0'),
),
'AddColumnsProcessedMetrics' => array(
'filter_add_columns_when_show_all_columns' => array('integer')
),
'AddColumnsProcessedMetricsGoal' => array(
'filter_update_columns_when_show_all_goals' => array('integer'),
'idGoal' => array('string', AddColumnsProcessedMetricsGoal::GOALS_OVERVIEW),
),
'Sort' => array(
'filter_sort_column' => array('string'),
'filter_sort_order' => array('string', 'desc'),
),
'Truncate' => array(
'filter_truncate' => array('integer'),
),
'Limit' => array(
'filter_offset' => array('integer', '0'),
'filter_limit' => array('integer'),
'keep_summary_row' => array('integer', '0'),
),
);
return array(
array('Pattern',
array(
'filter_column' => array('string', 'label'),
'filter_pattern' => array('string')
)),
array('PatternRecursive',
array(
'filter_column_recursive' => array('string', 'label'),
'filter_pattern_recursive' => array('string'),
)),
array('ExcludeLowPopulation',
array(
'filter_excludelowpop' => array('string'),
'filter_excludelowpop_value' => array('float', '0'),
)),
array('Sort',
array(
'filter_sort_column' => array('string'),
'filter_sort_order' => array('string', 'desc'),
)),
array('Truncate',
array(
'filter_truncate' => array('integer'),
)),
array('Limit',
array(
'filter_offset' => array('integer', '0'),
'filter_limit' => array('integer'),
'keep_summary_row' => array('integer', '0'),
))
);
}
private function getGenericFiltersHavingDefaultValues()
{
$filters = self::getGenericFiltersInformation();
if ($this->report && $this->report->getDefaultSortColumn()) {
foreach ($filters as $index => $filter) {
if ($filter[0] === 'Sort') {
$filters[$index][1]['filter_sort_column'] = array('string', $this->report->getDefaultSortColumn());
$filters[$index][1]['filter_sort_order'] = array('string', $this->report->getDefaultSortOrder());
}
}
}
return self::$genericFiltersInfo;
return $filters;
}
/**
@ -107,13 +145,20 @@ class DataTableGenericFilter
return;
}
$genericFilters = self::getGenericFiltersInformation();
$genericFilters = $this->getGenericFiltersHavingDefaultValues();
$filterApplied = false;
foreach ($genericFilters as $filterName => $parameters) {
foreach ($genericFilters as $filterMeta) {
$filterName = $filterMeta[0];
$filterParams = $filterMeta[1];
$filterParameters = array();
$exceptionRaised = false;
foreach ($parameters as $name => $info) {
if (in_array($filterName, $this->disabledFilters)) {
continue;
}
foreach ($filterParams as $name => $info) {
// parameter type to cast to
$type = $info[0];
@ -123,12 +168,6 @@ class DataTableGenericFilter
$defaultValue = $info[1];
}
// third element in the array, if it exists, overrides the name of the request variable
$varName = $name;
if (isset($info[2])) {
$varName = $info[2];
}
try {
$value = Common::getRequestVar($name, $defaultValue, $type, $this->request);
settype($value, $type);
@ -144,6 +183,45 @@ class DataTableGenericFilter
$filterApplied = true;
}
}
return $filterApplied;
}
public function areProcessedMetricsNeededFor($metrics)
{
$columnQueryParameters = array(
'filter_column',
'filter_column_recursive',
'filter_excludelowpop',
'filter_sort_column'
);
foreach ($columnQueryParameters as $queryParamName) {
$queryParamValue = Common::getRequestVar($queryParamName, false, $type = null, $this->request);
if (!empty($queryParamValue)
&& $this->containsProcessedMetric($metrics, $queryParamValue)
) {
return true;
}
}
return false;
}
/**
* @param ProcessedMetric[] $metrics
* @param string $name
* @return bool
*/
private function containsProcessedMetric($metrics, $name)
{
foreach ($metrics as $metric) {
if ($metric instanceof ProcessedMetric
&& $metric->getName() == $name
) {
return true;
}
}
return false;
}
}

44
www/analytics/core/API/DataTableManipulator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,7 +10,6 @@ namespace Piwik\API;
use Exception;
use Piwik\Archive\DataTableFactory;
use Piwik\Common;
use Piwik\DataTable\Row;
use Piwik\DataTable;
use Piwik\Period\Range;
@ -63,7 +62,7 @@ abstract class DataTableManipulator
{
if ($dataTable instanceof DataTable\Map) {
return $this->manipulateDataTableMap($dataTable);
} else if ($dataTable instanceof DataTable) {
} elseif ($dataTable instanceof DataTable) {
return $this->manipulateDataTable($dataTable);
} else {
return $dataTable;
@ -90,7 +89,7 @@ abstract class DataTableManipulator
* Manipulates a single DataTable instance. Derived classes must define
* this function.
*/
protected abstract function manipulateDataTable($dataTable);
abstract protected function manipulateDataTable($dataTable);
/**
* Load the subtable for a row.
@ -124,7 +123,7 @@ abstract class DataTableManipulator
}
}
$method = $this->getApiMethodForSubtable();
$method = $this->getApiMethodForSubtable($request);
return $this->callApiAndReturnDataTable($this->apiModule, $method, $request);
}
@ -136,19 +135,34 @@ abstract class DataTableManipulator
* @param $request
* @return
*/
protected abstract function manipulateSubtableRequest($request);
abstract protected function manipulateSubtableRequest($request);
/**
* Extract the API method for loading subtables from the meta data
*
* @throws Exception
* @return string
*/
private function getApiMethodForSubtable()
private function getApiMethodForSubtable($request)
{
if (!$this->apiMethodForSubtable) {
$meta = API::getInstance()->getMetadata('all', $this->apiModule, $this->apiMethod);
if (!empty($request['idSite'])) {
$idSite = $request['idSite'];
} else {
$idSite = 'all';
}
if(empty($meta)) {
$apiParameters = array();
if (!empty($request['idDimension'])) {
$apiParameters['idDimension'] = $request['idDimension'];
}
if (!empty($request['idGoal'])) {
$apiParameters['idGoal'] = $request['idGoal'];
}
$meta = API::getInstance()->getMetadata($idSite, $this->apiModule, $this->apiMethod, $apiParameters);
if (empty($meta)) {
throw new Exception(sprintf(
"The DataTable cannot be manipulated: Metadata for report %s.%s could not be found. You can define the metadata in a hook, see example at: http://developer.piwik.org/api-reference/events#apigetreportmetadata",
$this->apiModule, $this->apiMethod
@ -171,6 +185,8 @@ abstract class DataTableManipulator
$request = $this->manipulateSubtableRequest($request);
$request['serialize'] = 0;
$request['expanded'] = 0;
$request['format'] = 'original';
$request['format_metrics'] = 0;
// don't want to run recursive filters on the subtables as they are loaded,
// otherwise the result will be empty in places (or everywhere). instead we
@ -179,14 +195,8 @@ abstract class DataTableManipulator
$dataTable = Proxy::getInstance()->call($class, $method, $request);
$response = new ResponseBuilder($format = 'original', $request);
$dataTable = $response->getResponse($dataTable);
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $request) == 0) {
if (method_exists($dataTable, 'applyQueuedFilters')) {
$dataTable->applyQueuedFilters();
}
}
$response->disableSendHeader();
$dataTable = $response->getResponse($dataTable, $apiModule, $method);
return $dataTable;
}
}

44
www/analytics/core/API/DataTableManipulator/Flattener.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -38,17 +38,16 @@ class Flattener extends DataTableManipulator
* Separator for building recursive labels (or paths)
* @var string
*/
public $recursiveLabelSeparator = ' - ';
public $recursiveLabelSeparator = '';
/**
* @param DataTable $dataTable
* @param string $recursiveLabelSeparator
* @return DataTable|DataTable\Map
*/
public function flatten($dataTable)
public function flatten($dataTable, $recursiveLabelSeparator)
{
if ($this->apiModule == 'Actions' || $this->apiMethod == 'getWebsites') {
$this->recursiveLabelSeparator = '/';
}
$this->recursiveLabelSeparator = $recursiveLabelSeparator;
return $this->manipulate($dataTable);
}
@ -72,9 +71,10 @@ class Flattener extends DataTableManipulator
}
$newDataTable = $dataTable->getEmptyClone($keepFilters);
foreach ($dataTable->getRows() as $row) {
$this->flattenRow($row, $newDataTable);
foreach ($dataTable->getRows() as $rowId => $row) {
$this->flattenRow($row, $rowId, $newDataTable);
}
return $newDataTable;
}
@ -84,15 +84,21 @@ class Flattener extends DataTableManipulator
* @param string $labelPrefix
* @param bool $parentLogo
*/
private function flattenRow(Row $row, DataTable $dataTable,
private function flattenRow(Row $row, $rowId, DataTable $dataTable,
$labelPrefix = '', $parentLogo = false)
{
$label = $row->getColumn('label');
if ($label !== false) {
$label = trim($label);
if (substr($label, 0, 1) == '/' && $this->recursiveLabelSeparator == '/') {
$label = substr($label, 1);
if ($this->recursiveLabelSeparator == '/') {
if (substr($label, 0, 1) == '/') {
$label = substr($label, 1);
} elseif ($rowId === DataTable::ID_SUMMARY_ROW && $labelPrefix && $label != DataTable::LABEL_SUMMARY_ROW) {
$label = ' - ' . $label;
}
}
$label = $labelPrefix . $label;
$row->setColumn('label', $label);
}
@ -103,7 +109,16 @@ class Flattener extends DataTableManipulator
$row->setMetadata('logo', $logo);
}
$subTable = $this->loadSubtable($dataTable, $row);
/** @var DataTable $subTable */
$subTable = $row->getSubtable();
if ($subTable) {
$subTable->applyQueuedFilters();
$row->deleteMetadata('idsubdatatable_in_db');
} else {
$subTable = $this->loadSubtable($dataTable, $row);
}
$row->removeSubtable();
if ($subTable === null) {
@ -117,8 +132,8 @@ class Flattener extends DataTableManipulator
$dataTable->addRow($row);
}
$prefix = $label . $this->recursiveLabelSeparator;
foreach ($subTable->getRows() as $row) {
$this->flattenRow($row, $dataTable, $prefix, $logo);
foreach ($subTable->getRows() as $rowId => $row) {
$this->flattenRow($row, $rowId, $dataTable, $prefix, $logo);
}
}
}
@ -127,6 +142,7 @@ class Flattener extends DataTableManipulator
* Remove the flat parameter from the subtable request
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{

39
www/analytics/core/API/DataTableManipulator/LabelFilter.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -24,6 +24,7 @@ use Piwik\DataTable\Row;
class LabelFilter extends DataTableManipulator
{
const SEPARATOR_RECURSIVE_LABEL = '>';
const TERMINAL_OPERATOR = '@';
private $labels;
private $addLabelIndex;
@ -63,6 +64,10 @@ class LabelFilter extends DataTableManipulator
*/
private function doFilterRecursiveDescend($labelParts, $dataTable)
{
// we need to make sure to rebuild the index as some filters change the label column directly via
// $row->setColumn('label', '') which would not be noticed in the label index otherwise.
$dataTable->rebuildIndex();
// search for the first part of the tree search
$labelPart = array_shift($labelParts);
@ -101,6 +106,9 @@ class LabelFilter extends DataTableManipulator
protected function manipulateSubtableRequest($request)
{
unset($request['label']);
unset($request['flat']);
$request['totals'] = 0;
$request['filter_sort_column'] = ''; // do not sort, we only want to find a matching column
return $request;
}
@ -111,16 +119,22 @@ class LabelFilter extends DataTableManipulator
* Note: The HTML Encoded version must be tried first, since in ResponseBuilder the $label is unsanitized
* via Common::unsanitizeLabelParameter.
*
* @param string $label
* @param string $originalLabel
* @return array
*/
private function getLabelVariations($label)
private function getLabelVariations($originalLabel)
{
static $pageTitleReports = array('getPageTitles', 'getEntryPageTitles', 'getExitPageTitles');
$originalLabel = trim($originalLabel);
$isTerminal = substr($originalLabel, 0, 1) == self::TERMINAL_OPERATOR;
if ($isTerminal) {
$originalLabel = substr($originalLabel, 1);
}
$variations = array();
$label = urldecode($label);
$label = trim($label);
$label = trim(urldecode($originalLabel));
$sanitizedLabel = Common::sanitizeInputValue($label);
$variations[] = $sanitizedLabel;
@ -128,13 +142,20 @@ class LabelFilter extends DataTableManipulator
if ($this->apiModule == 'Actions'
&& in_array($this->apiMethod, $pageTitleReports)
) {
// special case: the Actions.getPageTitles report prefixes some labels with a blank.
// the blank might be passed by the user but is removed in Request::getRequestArrayFromString.
$variations[] = ' ' . $sanitizedLabel;
$variations[] = ' ' . $label;
if ($isTerminal) {
array_unshift($variations, ' ' . $sanitizedLabel);
array_unshift($variations, ' ' . $label);
} else {
// special case: the Actions.getPageTitles report prefixes some labels with a blank.
// the blank might be passed by the user but is removed in Request::getRequestArrayFromString.
$variations[] = ' ' . $sanitizedLabel;
$variations[] = ' ' . $label;
}
}
$variations[] = $label;
$variations = array_unique($variations);
return $variations;
}

189
www/analytics/core/API/DataTableManipulator/ReportTotalsCalculator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,13 +10,9 @@ namespace Piwik\API\DataTableManipulator;
use Piwik\API\DataTableManipulator;
use Piwik\DataTable;
use Piwik\DataTable\Row;
use Piwik\DataTable\BaseFilter;
use Piwik\Period\Range;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\Metrics;
use Piwik\Plugins\API\API;
use Piwik\Period;
use Piwik\Plugin\Report;
/**
* This class is responsible for setting the metadata property 'totals' on each dataTable if the report
@ -26,10 +22,29 @@ use Piwik\Plugins\API\API;
class ReportTotalsCalculator extends DataTableManipulator
{
/**
* Cached report metadata array.
* Array [readableMetric] => [summed value]
* @var array
*/
private static $reportMetadata = array();
private $totals = array();
/**
* @var Report
*/
private $report;
/**
* Constructor
*
* @param bool $apiModule
* @param bool $apiMethod
* @param array $request
* @param Report $report
*/
public function __construct($apiModule = false, $apiMethod = false, $request = array(), $report = null)
{
parent::__construct($apiModule, $apiMethod, $request);
$this->report = $report;
}
/**
* @param DataTable $table
@ -46,7 +61,7 @@ class ReportTotalsCalculator extends DataTableManipulator
try {
return $this->manipulate($table);
} catch(\Exception $e) {
} catch (\Exception $e) {
// eg. requests with idSubtable may trigger this exception
// (where idSubtable was removed in
// ?module=API&method=Events.getNameFromCategoryId&idSubtable=1&secondaryDimension=eventName&format=XML&idSite=1&period=day&date=yesterday&flat=0
@ -62,75 +77,32 @@ class ReportTotalsCalculator extends DataTableManipulator
*/
protected function manipulateDataTable($dataTable)
{
$report = $this->findCurrentReport();
if (!empty($report) && empty($report['dimension'])) {
if (!empty($this->report) && !$this->report->getDimension() && !$this->isAllMetricsReport()) {
// we currently do not calculate the total value for reports having no dimension
return $dataTable;
}
// Array [readableMetric] => [summed value]
$totalValues = array();
$this->totals = array();
$firstLevelTable = $this->makeSureToWorkOnFirstLevelDataTable($dataTable);
$metricsToCalculate = Metrics::getMetricIdsToProcessReportTotal();
$metricNames = array();
foreach ($metricsToCalculate as $metricId) {
if (!$this->hasDataTableMetric($firstLevelTable, $metricId)) {
continue;
}
$metricNames[$metricId] = Metrics::getReadableColumnName($metricId);
}
foreach ($firstLevelTable->getRows() as $row) {
$totalValues = $this->sumColumnValueToTotal($row, $metricId, $totalValues);
foreach ($firstLevelTable->getRows() as $row) {
$columns = $row->getColumns();
foreach ($metricNames as $metricId => $metricName) {
$this->sumColumnValueToTotal($columns, $metricId, $metricName);
}
}
$dataTable->setMetadata('totals', $totalValues);
$dataTable->setMetadata('totals', $this->totals);
return $dataTable;
}
private function hasDataTableMetric(DataTable $dataTable, $metricId)
{
$firstRow = $dataTable->getFirstRow();
if (empty($firstRow)) {
return false;
}
if (false === $this->getColumn($firstRow, $metricId)) {
return false;
}
return true;
}
/**
* Returns column from a given row.
* Will work with 2 types of datatable
* - raw datatables coming from the archive DB, which columns are int indexed
* - datatables processed resulting of API calls, which columns have human readable english names
*
* @param Row|array $row
* @param int $columnIdRaw see consts in Metrics::
* @return mixed Value of column, false if not found
*/
private function getColumn($row, $columnIdRaw)
{
$columnIdReadable = Metrics::getReadableColumnName($columnIdRaw);
if ($row instanceof Row) {
$raw = $row->getColumn($columnIdRaw);
if ($raw !== false) {
return $raw;
}
return $row->getColumn($columnIdReadable);
}
return false;
}
private function makeSureToWorkOnFirstLevelDataTable($table)
{
if (!array_key_exists('idSubtable', $this->request)) {
@ -144,8 +116,8 @@ class ReportTotalsCalculator extends DataTableManipulator
$module = $this->apiModule;
$action = $this->apiMethod;
} else {
$module = $firstLevelReport['module'];
$action = $firstLevelReport['action'];
$module = $firstLevelReport->getModule();
$action = $firstLevelReport->getAction();
}
$request = $this->request;
@ -164,33 +136,56 @@ class ReportTotalsCalculator extends DataTableManipulator
}
}
return $this->callApiAndReturnDataTable($module, $action, $request);
$table = $this->callApiAndReturnDataTable($module, $action, $request);
if ($table instanceof DataTable\Map) {
$table = $table->mergeChildren();
}
return $table;
}
private function sumColumnValueToTotal(Row $row, $metricId, $totalValues)
private function sumColumnValueToTotal($columns, $metricId, $metricName)
{
$value = $this->getColumn($row, $metricId);
if (false === $value) {
return $totalValues;
$value = false;
if (array_key_exists($metricId, $columns)) {
$value = $columns[$metricId];
}
$metricName = Metrics::getReadableColumnName($metricId);
if ($value === false) {
// we do not add $metricId to $possibleMetricNames for a small performance improvement since in most cases
// $metricId should be present in $columns so we avoid this foreach loop
$possibleMetricNames = array(
$metricName,
// TODO: this and below is a hack to get report totals to work correctly w/ MultiSites.getAll. can be corrected
// when all metrics are described by Metadata classes & internal naming quirks are handled by core system.
'Goal_' . $metricName,
'Actions_' . $metricName
);
foreach ($possibleMetricNames as $possibleMetricName) {
if (array_key_exists($possibleMetricName, $columns)) {
$value = $columns[$possibleMetricName];
break;
}
}
if (array_key_exists($metricName, $totalValues)) {
$totalValues[$metricName] += $value;
if ($value === false) {
return;
}
}
if (array_key_exists($metricName, $this->totals)) {
$this->totals[$metricName] += $value;
} else {
$totalValues[$metricName] = $value;
$this->totals[$metricName] = $value;
}
return $totalValues;
}
/**
* Make sure to get all rows of the first level table.
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{
@ -198,6 +193,7 @@ class ReportTotalsCalculator extends DataTableManipulator
$request['expanded'] = 0;
$request['filter_limit'] = -1;
$request['filter_offset'] = 0;
$request['filter_sort_column'] = '';
$parametersToRemove = array('flat');
@ -213,38 +209,21 @@ class ReportTotalsCalculator extends DataTableManipulator
return $request;
}
private function getReportMetadata()
{
if (!empty(static::$reportMetadata)) {
return static::$reportMetadata;
}
static::$reportMetadata = API::getInstance()->getReportMetadata();
return static::$reportMetadata;
}
private function findCurrentReport()
{
foreach ($this->getReportMetadata() as $report) {
if ($this->apiMethod == $report['action']
&& $this->apiModule == $report['module']) {
return $report;
}
}
}
private function findFirstLevelReport()
{
foreach ($this->getReportMetadata() as $report) {
if (!empty($report['actionToLoadSubTables'])
&& $this->apiMethod == $report['actionToLoadSubTables']
&& $this->apiModule == $report['module']
foreach (Report::getAllReports() as $report) {
$actionToLoadSubtables = $report->getActionToLoadSubTables();
if ($actionToLoadSubtables == $this->apiMethod
&& $this->apiModule == $report->getModule()
) {
return $report;
}
}
return null;
}
private function isAllMetricsReport()
{
return $this->report->getModule() == 'API' && $this->report->getAction() == 'get';
}
}

436
www/analytics/core/API/DataTablePostProcessor.php Normal file
View file

@ -0,0 +1,436 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
use Exception;
use Piwik\API\DataTableManipulator\Flattener;
use Piwik\API\DataTableManipulator\LabelFilter;
use Piwik\API\DataTableManipulator\ReportTotalsCalculator;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\PivotByDimension;
use Piwik\Metrics\Formatter;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
/**
* Processes DataTables that should be served through Piwik's APIs. This processing handles
* special query parameters and computes processed metrics. It does not included rendering to
* output formats (eg, 'xml').
*/
class DataTablePostProcessor
{
const PROCESSED_METRICS_COMPUTED_FLAG = 'processed_metrics_computed';
/**
* @var null|Report
*/
private $report;
/**
* @var string[]
*/
private $request;
/**
* @var string
*/
private $apiModule;
/**
* @var string
*/
private $apiMethod;
/**
* @var Inconsistencies
*/
private $apiInconsistencies;
/**
* @var Formatter
*/
private $formatter;
private $callbackBeforeGenericFilters;
private $callbackAfterGenericFilters;
/**
* Constructor.
*/
public function __construct($apiModule, $apiMethod, $request)
{
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
$this->setRequest($request);
$this->report = Report::factory($apiModule, $apiMethod);
$this->apiInconsistencies = new Inconsistencies();
$this->setFormatter(new Formatter());
}
public function setFormatter(Formatter $formatter)
{
$this->formatter = $formatter;
}
public function setRequest($request)
{
$this->request = $request;
}
public function setCallbackBeforeGenericFilters($callbackBeforeGenericFilters)
{
$this->callbackBeforeGenericFilters = $callbackBeforeGenericFilters;
}
public function setCallbackAfterGenericFilters($callbackAfterGenericFilters)
{
$this->callbackAfterGenericFilters = $callbackAfterGenericFilters;
}
/**
* Apply post-processing logic to a DataTable of a report for an API request.
*
* @param DataTableInterface $dataTable The data table to process.
* @return DataTableInterface A new data table.
*/
public function process(DataTableInterface $dataTable)
{
// TODO: when calculating metrics before hand, only calculate for needed metrics, not all. NOTE:
// this is non-trivial since it will require, eg, to make sure processed metrics aren't added
// after pivotBy is handled.
$dataTable = $this->applyPivotByFilter($dataTable);
$dataTable = $this->applyTotalsCalculator($dataTable);
$dataTable = $this->applyFlattener($dataTable);
if ($this->callbackBeforeGenericFilters) {
call_user_func($this->callbackBeforeGenericFilters, $dataTable);
}
$dataTable = $this->applyGenericFilters($dataTable);
$this->applyComputeProcessedMetrics($dataTable);
if ($this->callbackAfterGenericFilters) {
call_user_func($this->callbackAfterGenericFilters, $dataTable);
}
// we automatically safe decode all datatable labels (against xss)
$dataTable->queueFilter('SafeDecodeLabel');
$dataTable = $this->convertSegmentValueToSegment($dataTable);
$dataTable = $this->applyQueuedFilters($dataTable);
$dataTable = $this->applyRequestedColumnDeletion($dataTable);
$dataTable = $this->applyLabelFilter($dataTable);
$dataTable = $this->applyMetricsFormatting($dataTable);
return $dataTable;
}
private function convertSegmentValueToSegment(DataTableInterface $dataTable)
{
$dataTable->filter('AddSegmentBySegmentValue', array($this->report));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segmentValue'));
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyPivotByFilter(DataTableInterface $dataTable)
{
$pivotBy = Common::getRequestVar('pivotBy', false, 'string', $this->request);
if (!empty($pivotBy)) {
$this->applyComputeProcessedMetrics($dataTable);
$reportId = $this->apiModule . '.' . $this->apiMethod;
$pivotByColumn = Common::getRequestVar('pivotByColumn', false, 'string', $this->request);
$pivotByColumnLimit = Common::getRequestVar('pivotByColumnLimit', false, 'int', $this->request);
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segmentValue'));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segment'));
$dataTable->filter('PivotByDimension', array($reportId, $pivotBy, $pivotByColumn, $pivotByColumnLimit,
PivotByDimension::isSegmentFetchingEnabledInConfig()));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTable|DataTableInterface|DataTable\Map
*/
public function applyFlattener($dataTable)
{
if (Common::getRequestVar('flat', '0', 'string', $this->request) == '1') {
$flattener = new Flattener($this->apiModule, $this->apiMethod, $this->request);
if (Common::getRequestVar('include_aggregate_rows', '0', 'string', $this->request) == '1') {
$flattener->includeAggregateRows();
}
$recursiveLabelSeparator = ' - ';
if ($this->report) {
$recursiveLabelSeparator = $this->report->getRecursiveLabelSeparator();
}
$dataTable = $flattener->flatten($dataTable, $recursiveLabelSeparator);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyTotalsCalculator($dataTable)
{
if (1 == Common::getRequestVar('totals', '1', 'integer', $this->request)) {
$calculator = new ReportTotalsCalculator($this->apiModule, $this->apiMethod, $this->request, $this->report);
$dataTable = $calculator->calculate($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyGenericFilters($dataTable)
{
// if the flag disable_generic_filters is defined we skip the generic filters
if (0 == Common::getRequestVar('disable_generic_filters', '0', 'string', $this->request)) {
$this->applyProcessedMetricsGenericFilters($dataTable);
$genericFilter = new DataTableGenericFilter($this->request, $this->report);
$self = $this;
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($genericFilter, $report, $self) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
if ($genericFilter->areProcessedMetricsNeededFor($processedMetrics)) {
$self->computeProcessedMetrics($table);
}
});
$label = self::getLabelFromRequest($this->request);
if (!empty($label)) {
$genericFilter->disableFilters(array('Limit', 'Truncate'));
}
$genericFilter->filter($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyProcessedMetricsGenericFilters($dataTable)
{
$addNormalProcessedMetrics = null;
try {
$addNormalProcessedMetrics = Common::getRequestVar(
'filter_add_columns_when_show_all_columns', null, 'integer', $this->request);
} catch (Exception $ex) {
// ignore
}
if ($addNormalProcessedMetrics !== null) {
$dataTable->filter('AddColumnsProcessedMetrics', array($addNormalProcessedMetrics));
}
$addGoalProcessedMetrics = null;
try {
$addGoalProcessedMetrics = Common::getRequestVar(
'filter_update_columns_when_show_all_goals', null, 'integer', $this->request);
} catch (Exception $ex) {
// ignore
}
if ($addGoalProcessedMetrics !== null) {
$idGoal = Common::getRequestVar(
'idGoal', DataTable\Filter\AddColumnsProcessedMetricsGoal::GOALS_OVERVIEW, 'string', $this->request);
$dataTable->filter('AddColumnsProcessedMetricsGoal', array($ignore = true, $idGoal));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyQueuedFilters($dataTable)
{
// if the flag disable_queued_filters is defined we skip the filters that were queued
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $this->request) == 0) {
$dataTable->applyQueuedFilters();
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyRequestedColumnDeletion($dataTable)
{
// use the ColumnDelete filter if hideColumns/showColumns is provided (must be done
// after queued filters are run so processed metrics can be removed, too)
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
$showRawMetrics = Common::getRequestVar('showRawMetrics', 0, 'int', $this->request);
if (!empty($hideColumns)
|| !empty($showColumns)
) {
$dataTable->filter('ColumnDelete', array($hideColumns, $showColumns));
} else if ($showRawMetrics !== 1) {
$this->removeTemporaryMetrics($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
*/
public function removeTemporaryMetrics(DataTableInterface $dataTable)
{
$allColumns = !empty($this->report) ? $this->report->getAllMetrics() : array();
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($report, $allColumns) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
$allTemporaryMetrics = array();
foreach ($processedMetrics as $metric) {
$allTemporaryMetrics = array_merge($allTemporaryMetrics, $metric->getTemporaryMetrics());
}
if (!empty($allTemporaryMetrics)) {
$table->filter('ColumnDelete', array($allTemporaryMetrics));
}
});
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyLabelFilter($dataTable)
{
$label = self::getLabelFromRequest($this->request);
// apply label filter: only return rows matching the label parameter (more than one if more than one label)
if (!empty($label)) {
$addLabelIndex = Common::getRequestVar('labelFilterAddLabelIndex', 0, 'int', $this->request) == 1;
$filter = new LabelFilter($this->apiModule, $this->apiMethod, $this->request);
$dataTable = $filter->filter($label, $dataTable, $addLabelIndex);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyMetricsFormatting($dataTable)
{
$formatMetrics = Common::getRequestVar('format_metrics', 0, 'string', $this->request);
if ($formatMetrics == '0') {
return $dataTable;
}
// in Piwik 2.X & below, metrics are not formatted in API responses except for percents.
// this code implements this inconsistency
$onlyFormatPercents = $formatMetrics === 'bc';
$metricsToFormat = null;
if ($onlyFormatPercents) {
$metricsToFormat = $this->apiInconsistencies->getPercentMetricsToFormat();
}
$dataTable->filter(array($this->formatter, 'formatMetrics'), array($this->report, $metricsToFormat));
return $dataTable;
}
/**
* Returns the value for the label query parameter which can be either a string
* (ie, label=...) or array (ie, label[]=...).
*
* @param array $request
* @return array
*/
public static function getLabelFromRequest($request)
{
$label = Common::getRequestVar('label', array(), 'array', $request);
if (empty($label)) {
$label = Common::getRequestVar('label', '', 'string', $request);
if (!empty($label)) {
$label = array($label);
}
}
$label = self::unsanitizeLabelParameter($label);
return $label;
}
public static function unsanitizeLabelParameter($label)
{
// this is needed because Proxy uses Common::getRequestVar which in turn
// uses Common::sanitizeInputValue. This causes the > that separates recursive labels
// to become &gt; and we need to undo that here.
$label = str_replace( htmlentities('>'), '>', $label);
return $label;
}
public function computeProcessedMetrics(DataTable $dataTable)
{
if ($dataTable->getMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG)) {
return;
}
/** @var ProcessedMetric[] $processedMetrics */
$processedMetrics = Report::getProcessedMetricsForTable($dataTable, $this->report);
if (empty($processedMetrics)) {
return;
}
$dataTable->setMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG, true);
foreach ($processedMetrics as $name => $processedMetric) {
if (!$processedMetric->beforeCompute($this->report, $dataTable)) {
continue;
}
foreach ($dataTable->getRows() as $row) {
if ($row->getColumn($name) === false) { // only compute the metric if it has not been computed already
$computedValue = $processedMetric->compute($row);
if ($computedValue !== false) {
$row->addColumn($name, $computedValue);
}
$subtable = $row->getSubtable();
if (!empty($subtable)) {
$this->computeProcessedMetrics($subtable);
}
}
}
}
}
public function applyComputeProcessedMetrics(DataTableInterface $dataTable)
{
$dataTable->filter(array($this, 'computeProcessedMetrics'));
}
}

242
www/analytics/core/API/DocumentationGenerator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -12,10 +12,10 @@ use Exception;
use Piwik\Common;
use Piwik\Piwik;
use Piwik\Url;
use ReflectionClass;
class DocumentationGenerator
{
protected $modulesToHide = array('CoreAdminHome', 'DBStats');
protected $countPluginsLoaded = 0;
/**
@ -47,63 +47,152 @@ class DocumentationGenerator
if (!empty($prefixUrls)) {
$prefixUrls = 'http://demo.piwik.org/';
}
$str = $toc = '';
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$parametersToSet = array(
'idSite' => Common::getRequestVar('idSite', 1, 'int'),
'period' => Common::getRequestVar('period', 'day', 'string'),
'date' => Common::getRequestVar('date', 'today', 'string')
);
foreach (Proxy::getInstance()->getMetadata() as $class => $info) {
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
if (in_array($moduleName, $this->modulesToHide)) {
$rClass = new ReflectionClass($class);
if (!Piwik::hasUserSuperUserAccess() && $this->checkIfClassCommentContainsHideAnnotation($rClass)) {
continue;
}
$toc .= "<a href='#$moduleName'>$moduleName</a><br/>";
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
// we prefix all URLs with $prefixUrls
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss1 = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$exampleUrlRss2 = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last5', 'period' => 'week',) + $parametersToSet);
$lastNUrls = ", RSS of the last <a target=_blank href='$exampleUrlRss1&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target=_blank href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target=_blank href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target=_blank href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
}
$str .= '</small>';
$str .= "</div>\n";
$toDisplay = $this->prepareModulesAndMethods($info, $moduleName);
foreach ($toDisplay as $moduleName => $methods) {
$toc .= $this->prepareModuleToDisplay($moduleName);
$str .= $this->prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls);
}
$str .= '<div style="margin:15px;"><a href="#topApiRef">↑ Back to top</a></div>';
}
$str = "<h2 id='topApiRef' name='topApiRef'>Quick access to APIs</h2>
$toc
$str";
return $str;
}
public function prepareModuleToDisplay($moduleName)
{
return "<a href='#$moduleName'>$moduleName</a><br/>";
}
public function prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls)
{
$str = '';
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$info['__documentation'] = $this->checkDocumentation($info['__documentation']);
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($methods as $methodName) {
if (Proxy::getInstance()->isDeprecatedMethod($class, $methodName)) {
continue;
}
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
$str .= $this->addExamples($class, $methodName, $prefixUrls);
}
$str .= '</small>';
$str .= "</div>\n";
}
return $str;
}
public function prepareModulesAndMethods($info, $moduleName)
{
$toDisplay = array();
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
$toDisplay[$moduleName][] = $methodName;
}
return $toDisplay;
}
public function addExamples($class, $methodName, $prefixUrls)
{
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$parametersToSet = array(
'idSite' => Common::getRequestVar('idSite', 1, 'int'),
'period' => Common::getRequestVar('period', 'day', 'string'),
'date' => Common::getRequestVar('date', 'today', 'string')
);
$str = '';
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$lastNUrls = ", RSS of the last <a target='_blank' href='$exampleUrlRss&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target='_blank' href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target='_blank' href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target='_blank' href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
return $str;
}
/**
* Check if Class contains @hide
*
* @param ReflectionClass $rClass instance of ReflectionMethod
* @return bool
*/
public function checkIfClassCommentContainsHideAnnotation(ReflectionClass $rClass)
{
return false !== strstr($rClass->getDocComment(), '@hide');
}
/**
* Check if documentation contains @hide annotation and deletes it
*
* @param $moduleToCheck
* @return mixed
*/
public function checkDocumentation($moduleToCheck)
{
if (strpos($moduleToCheck, '@hide') == true) {
$moduleToCheck = str_replace(strtok(strstr($moduleToCheck, '@hide'), "\n"), "", $moduleToCheck);
}
return $moduleToCheck;
}
private function getInterfaceString($moduleName, $class, $info, $parametersToSet, $outputExampleUrls, $prefixUrls)
{
$str = '';
$str .= "\n<a name='$moduleName' id='$moduleName'></a><h2>Module " . $moduleName . "</h2>";
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
if (Proxy::getInstance()->isDeprecatedMethod($class, $methodName)) {
continue;
}
$str .= $this->getMethodString($moduleName, $class, $parametersToSet, $outputExampleUrls, $prefixUrls, $methodName, $str);
}
$str .= '<div style="margin:15px;"><a href="#topApiRef">↑ Back to top</a></div>';
return $str;
}
@ -136,6 +225,7 @@ class DocumentationGenerator
'ip' => '194.57.91.215',
'idSites' => '1,2',
'idAlert' => '1',
'seconds' => '3600',
// 'segmentName' => 'browserCode',
);
@ -169,8 +259,8 @@ class DocumentationGenerator
$aParameters = Proxy::getInstance()->getParametersList($class, $methodName);
// Kindly force some known generic parameters to appear in the final list
// the parameter 'format' can be set to all API methods (used in tests)
// the parameter 'hideIdSubDatable' is used for integration tests only
// the parameter 'serialize' sets php outputs human readable, used in integration tests and debug
// the parameter 'hideIdSubDatable' is used for system tests only
// the parameter 'serialize' sets php outputs human readable, used in system tests and debug
// the parameter 'language' sets the language for the response (eg. country names)
// the parameter 'flat' reduces a hierarchical table to a single level by concatenating labels
// the parameter 'include_aggregate_rows' can be set to include inner nodes in flat reports
@ -183,12 +273,25 @@ class DocumentationGenerator
$aParameters['label'] = false;
$aParameters['flat'] = false;
$aParameters['include_aggregate_rows'] = false;
$aParameters['filter_limit'] = false; //@review without adding this, I can not set filter_limit in $otherRequestParameters integration tests
$aParameters['filter_sort_column'] = false; //@review without adding this, I can not set filter_sort_column in $otherRequestParameters integration tests
$aParameters['filter_offset'] = false; //@review without adding this, I can not set filter_offset in $otherRequestParameters system tests
$aParameters['filter_limit'] = false; //@review without adding this, I can not set filter_limit in $otherRequestParameters system tests
$aParameters['filter_sort_column'] = false; //@review without adding this, I can not set filter_sort_column in $otherRequestParameters system tests
$aParameters['filter_sort_order'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_excludelowpop'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_excludelowpop_value'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_column_recursive'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_pattern_recursive'] = false; //@review without adding this, I can not set filter_sort_order in $otherRequestParameters system tests
$aParameters['filter_truncate'] = false;
$aParameters['hideColumns'] = false;
$aParameters['showColumns'] = false;
$aParameters['filter_pattern_recursive'] = false;
$aParameters['pivotBy'] = false;
$aParameters['pivotByColumn'] = false;
$aParameters['pivotByColumnLimit'] = false;
$aParameters['disable_queued_filters'] = false;
$aParameters['disable_generic_filters'] = false;
$aParameters['expanded'] = false;
$aParameters['idDimenson'] = false;
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
$aParameters = array_merge(array('module' => 'API', 'method' => $moduleName . '.' . $methodName), $aParameters);
@ -235,4 +338,43 @@ class DocumentationGenerator
$sParameters = implode(", ", $asParameters);
return "($sParameters)";
}
private function getMethodString($moduleName, $class, $parametersToSet, $outputExampleUrls, $prefixUrls, $methodName)
{
$str = '';
$token_auth = "&token_auth=" . Piwik::getCurrentUserTokenAuth();
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
// we prefix all URLs with $prefixUrls
// used when we include this output in the Piwik official documentation for example
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$lastNUrls = ", RSS of the last <a target='_blank' href='$exampleUrlRss&format=rss$token_auth&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target='_blank' href='$exampleUrl&format=xml$token_auth'>XML</a>,
<a target='_blank' href='$exampleUrl&format=JSON$token_auth'>Json</a>,
<a target='_blank' href='$exampleUrl&format=Tsv$token_auth&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
}
$str .= '</small>';
$str .= "</div>\n";
return $str;
}
}

42
www/analytics/core/API/Inconsistencies.php Normal file
View file

@ -0,0 +1,42 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
/**
* Contains logic to replicate inconsistencies in Piwik's API. This class exists
* to provide a way to clean up existing Piwik code and behavior without breaking
* backwards compatibility immediately.
*
* Code that handles the case when the 'format_metrics' query parameter value is
* 'bc' should be removed as well. This code is in API\Request and DataTablePostProcessor.
*
* Should be removed before releasing Piwik 3.0.
*/
class Inconsistencies
{
/**
* In Piwik 2.X and below, the "raw" API would format percent values but no others.
* This method returns the list of percent metrics that were returned from the API
* formatted so we can maintain BC.
*
* Used by DataTablePostProcessor.
*/
public function getPercentMetricsToFormat()
{
return array(
'bounce_rate',
'conversion_rate',
'interaction_rate',
'exit_rate',
'bounce_rate_returning',
'nb_visits_percentage',
'/.*_evolution/',
'/goal_.*_conversion_rate/'
);
}
}

173
www/analytics/core/API/Proxy.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -24,7 +24,7 @@ use ReflectionMethod;
*
* It will also log the performance of API calls (time spent, parameter values, etc.) if logger available
*
* @method static \Piwik\API\Proxy getInstance()
* @method static Proxy getInstance()
*/
class Proxy extends Singleton
{
@ -37,10 +37,7 @@ class Proxy extends Singleton
// when a parameter doesn't have a default value we use this
private $noDefaultValue;
/**
* protected constructor
*/
protected function __construct()
public function __construct()
{
$this->noDefaultValue = new NoDefaultValue();
}
@ -78,12 +75,14 @@ class Proxy extends Singleton
$this->checkClassIsSingleton($className);
$rClass = new ReflectionClass($className);
foreach ($rClass->getMethods() as $method) {
$this->loadMethodMetadata($className, $method);
}
if (!$this->shouldHideAPIMethod($rClass->getDocComment())) {
foreach ($rClass->getMethods() as $method) {
$this->loadMethodMetadata($className, $method);
}
$this->setDocumentation($rClass, $className);
$this->alreadyRegistered[$className] = true;
$this->setDocumentation($rClass, $className);
$this->alreadyRegistered[$className] = true;
}
}
/**
@ -164,11 +163,11 @@ class Proxy extends Singleton
/**
* Triggered before an API request is dispatched.
*
*
* This event can be used to modify the arguments passed to one or more API methods.
*
*
* **Example**
*
*
* Piwik::addAction('API.Request.dispatch', function (&$parameters, $pluginName, $methodName) {
* if ($pluginName == 'Actions') {
* if ($methodName == 'getPageUrls') {
@ -178,7 +177,7 @@ class Proxy extends Singleton
* }
* }
* });
*
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
* @param string $pluginName The name of the plugin the API method belongs to.
* @param string $methodName The name of the API method that will be called.
@ -187,20 +186,20 @@ class Proxy extends Singleton
/**
* Triggered before an API request is dispatched.
*
*
* This event exists for convenience and is triggered directly after the {@hook API.Request.dispatch}
* event is triggered. It can be used to modify the arguments passed to a **single** API method.
*
*
* _Note: This is can be accomplished with the {@hook API.Request.dispatch} event as well, however
* event handlers for that event will have to do more work._
*
*
* **Example**
*
*
* Piwik::addAction('API.Actions.getPageUrls', function (&$parameters) {
* // force use of a single website. for some reason.
* $parameters['idSite'] = 1;
* });
*
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
*/
Piwik::postEvent(sprintf('API.%s.%s', $pluginName, $methodName), array(&$finalParameters));
@ -218,16 +217,16 @@ class Proxy extends Singleton
/**
* Triggered directly after an API request is dispatched.
*
*
* This event exists for convenience and is triggered immediately before the
* {@hook API.Request.dispatch.end} event. It can be used to modify the output of a **single**
* API method.
*
*
* _Note: This can be accomplished with the {@hook API.Request.dispatch.end} event as well,
* however event handlers for that event will have to do more work._
*
* **Example**
*
*
* // append (0 hits) to the end of row labels whose row has 0 hits
* Piwik::addAction('API.Actions.getPageUrls', function (&$returnValue, $info)) {
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
@ -238,13 +237,13 @@ class Proxy extends Singleton
* }
* }, null, array('nb_hits'));
* }
*
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* could be a {@link Piwik\DataTable DataTable}.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
@ -257,20 +256,20 @@ class Proxy extends Singleton
/**
* Triggered directly after an API request is dispatched.
*
*
* This event can be used to modify the output of any API method.
*
*
* **Example**
*
*
* // append (0 hits) to the end of row labels whose row has 0 hits for any report that has the 'nb_hits' metric
* Piwik::addAction('API.Actions.getPageUrls', function (&$returnValue, $info)) {
* Piwik::addAction('API.Actions.getPageUrls.end', function (&$returnValue, $info)) {
* // don't process non-DataTable reports and reports that don't have the nb_hits column
* if (!($returnValue instanceof DataTableInterface)
* || in_array('nb_hits', $returnValue->getColumns())
* ) {
* return;
* }
*
*
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
* if ($hits === 0) {
* return $label . " (0 hits)";
@ -279,12 +278,12 @@ class Proxy extends Singleton
* }
* }, null, array('nb_hits'));
* }
*
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
@ -323,6 +322,14 @@ class Proxy extends Singleton
return $this->metadataArray[$class][$name]['parameters'];
}
/**
* Check if given method name is deprecated or not.
*/
public function isDeprecatedMethod($class, $methodName)
{
return $this->metadataArray[$class][$methodName]['isDeprecated'];
}
/**
* Returns the 'moduleName' part of '\\Piwik\\Plugins\\moduleName\\API'
*
@ -378,7 +385,6 @@ class Proxy extends Singleton
$requestValue = Common::getRequestVar($name, null, null, $parametersRequest);
} else {
try {
if ($name == 'segment' && !empty($parametersRequest['segment'])) {
// segment parameter is an exception: we do not want to sanitize user input or it would break the segment encoding
$requestValue = ($parametersRequest['segment']);
@ -405,7 +411,7 @@ class Proxy extends Singleton
}
/**
* Includes the class API by looking up plugins/UserSettings/API.php
* Includes the class API by looking up plugins/xxx/API.php
*
* @param string $fileName api class name eg. "API"
* @throws Exception
@ -428,29 +434,27 @@ class Proxy extends Singleton
*/
private function loadMethodMetadata($class, $method)
{
if ($method->isPublic()
&& !$method->isConstructor()
&& $method->getName() != 'getInstance'
&& false === strstr($method->getDocComment(), '@deprecated')
&& (!$this->hideIgnoredFunctions || false === strstr($method->getDocComment(), '@ignore'))
) {
$name = $method->getName();
$parameters = $method->getParameters();
$aParameters = array();
foreach ($parameters as $parameter) {
$nameVariable = $parameter->getName();
$defaultValue = $this->noDefaultValue;
if ($parameter->isDefaultValueAvailable()) {
$defaultValue = $parameter->getDefaultValue();
}
$aParameters[$nameVariable] = $defaultValue;
}
$this->metadataArray[$class][$name]['parameters'] = $aParameters;
$this->metadataArray[$class][$name]['numberOfRequiredParameters'] = $method->getNumberOfRequiredParameters();
if (!$this->checkIfMethodIsAvailable($method)) {
return;
}
$name = $method->getName();
$parameters = $method->getParameters();
$docComment = $method->getDocComment();
$aParameters = array();
foreach ($parameters as $parameter) {
$nameVariable = $parameter->getName();
$defaultValue = $this->noDefaultValue;
if ($parameter->isDefaultValueAvailable()) {
$defaultValue = $parameter->getDefaultValue();
}
$aParameters[$nameVariable] = $defaultValue;
}
$this->metadataArray[$class][$name]['parameters'] = $aParameters;
$this->metadataArray[$class][$name]['numberOfRequiredParameters'] = $method->getNumberOfRequiredParameters();
$this->metadataArray[$class][$name]['isDeprecated'] = false !== strstr($docComment, '@deprecated');
}
/**
@ -468,15 +472,56 @@ class Proxy extends Singleton
}
/**
* Returns the number of required parameters (parameters without default values).
*
* @param string $class The class name
* @param string $name The method name
* @return int The number of required parameters
* @param $docComment
* @return bool
*/
private function getNumberOfRequiredParameters($class, $name)
public function shouldHideAPIMethod($docComment)
{
return $this->metadataArray[$class][$name]['numberOfRequiredParameters'];
$hideLine = strstr($docComment, '@hide');
if ($hideLine === false) {
return false;
}
$hideLine = trim($hideLine);
$hideLine .= ' ';
$token = trim(strtok($hideLine, " "), "\n");
$hide = false;
if (!empty($token)) {
/**
* This event exists for checking whether a Plugin API class or a Plugin API method tagged
* with a `@hideXYZ` should be hidden in the API listing.
*
* @param bool &$hide whether to hide APIs tagged with $token should be displayed.
*/
Piwik::postEvent(sprintf('API.DocumentationGenerator.%s', $token), array(&$hide));
}
return $hide;
}
/**
* @param ReflectionMethod $method
* @return bool
*/
protected function checkIfMethodIsAvailable(ReflectionMethod $method)
{
if (!$method->isPublic() || $method->isConstructor() || $method->getName() === 'getInstance') {
return false;
}
if ($this->hideIgnoredFunctions && false !== strstr($method->getDocComment(), '@ignore')) {
return false;
}
if ($this->shouldHideAPIMethod($method->getDocComment())) {
return false;
}
return true;
}
/**
@ -500,7 +545,7 @@ class Proxy extends Singleton
private function checkClassIsSingleton($className)
{
if (!method_exists($className, "getInstance")) {
throw new Exception("$className that provide an API must be Singleton and have a 'static public function getInstance()' method.");
throw new Exception("$className that provide an API must be Singleton and have a 'public static function getInstance()' method.");
}
}
}

274
www/analytics/core/API/Request.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -17,49 +17,49 @@ use Piwik\PluginDeactivatedException;
use Piwik\SettingsServer;
use Piwik\Url;
use Piwik\UrlHelper;
use Piwik\Log;
use Piwik\Plugin\Manager as PluginManager;
/**
* Dispatches API requests to the appropriate API method.
*
*
* The Request class is used throughout Piwik to call API methods. The difference
* between using Request and calling API methods directly is that Request
* will do more after calling the API including: applying generic filters, applying queued filters,
* and handling the **flat** and **label** query parameters.
*
*
* Additionally, the Request class will **forward current query parameters** to the request
* which is more convenient than calling {@link Piwik\Common::getRequestVar()} many times over.
*
*
* In most cases, using a Request object to query the API is the correct approach.
*
* ### Post-processing
*
*
* The return value of API methods undergo some extra processing before being returned by Request.
* To learn more about what happens to API results, read [this](/guides/piwiks-web-api#extra-report-processing).
*
* ### Output Formats
*
*
* The value returned by Request will be serialized to a certain format before being returned.
* To see the list of supported output formats, read [this](/guides/piwiks-web-api#output-formats).
*
*
* ### Examples
*
*
* **Basic Usage**
*
* $request = new Request('method=UserSettings.getWideScreen&idSite=1&date=yesterday&period=week'
*
* $request = new Request('method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week'
* . '&format=xml&filter_limit=5&filter_offset=0')
* $result = $request->process();
* echo $result;
*
*
* **Getting a unrendered DataTable**
*
*
* // use the convenience method 'processRequest'
* $dataTable = Request::processRequest('UserSettings.getWideScreen', array(
* $dataTable = Request::processRequest('UserLanguage.getLanguage', array(
* 'idSite' => 1,
* 'date' => 'yesterday',
* 'period' => 'week',
* 'filter_limit' => 5,
* 'filter_offset' => 0
*
*
* 'format' => 'original', // this is the important bit
* ));
* echo "This DataTable has " . $dataTable->getRowsCount() . " rows.";
@ -69,41 +69,46 @@ use Piwik\UrlHelper;
*/
class Request
{
protected $request = null;
private $request = null;
/**
* Converts the supplied request string into an array of query paramater name/value
* mappings. The current query parameters (everything in `$_GET` and `$_POST`) are
* forwarded to request array before it is returned.
*
* @param string|array $request The base request string or array, eg,
* `'module=UserSettings&action=getWidescreen'`.
* @param string|array|null $request The base request string or array, eg,
* `'module=UserLanguage&action=getLanguage'`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
* @return array
*/
static public function getRequestArrayFromString($request)
public static function getRequestArrayFromString($request, $defaultRequest = null)
{
$defaultRequest = $_GET + $_POST;
if ($defaultRequest === null) {
$defaultRequest = self::getDefaultRequest();
$requestRaw = self::getRequestParametersGET();
if (!empty($requestRaw['segment'])) {
$defaultRequest['segment'] = $requestRaw['segment'];
$requestRaw = self::getRequestParametersGET();
if (!empty($requestRaw['segment'])) {
$defaultRequest['segment'] = $requestRaw['segment'];
}
if (!isset($defaultRequest['format_metrics'])) {
$defaultRequest['format_metrics'] = 'bc';
}
}
$requestArray = $defaultRequest;
if (!is_null($request)) {
if (is_array($request)) {
$url = array();
foreach ($request as $key => $value) {
$url[] = $key . "=" . $value;
}
$request = implode("&", $url);
$requestParsed = $request;
} else {
$request = trim($request);
$request = str_replace(array("\n", "\t"), '', $request);
$requestParsed = UrlHelper::getArrayFromQueryString($request);
}
$request = trim($request);
$request = str_replace(array("\n", "\t"), '', $request);
$requestParsed = UrlHelper::getArrayFromQueryString($request);
$requestArray = $requestParsed + $defaultRequest;
}
@ -119,14 +124,17 @@ class Request
* Constructor.
*
* @param string|array $request Query string that defines the API call (must at least contain a **method** parameter),
* eg, `'method=UserSettings.getWideScreen&idSite=1&date=yesterday&period=week&format=xml'`
* eg, `'method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week&format=xml'`
* If a request is not provided, then we use the values in the `$_GET` and `$_POST`
* superglobals.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*/
public function __construct($request = null)
public function __construct($request = null, $defaultRequest = null)
{
$this->request = self::getRequestArrayFromString($request);
$this->request = self::getRequestArrayFromString($request, $defaultRequest);
$this->sanitizeRequest();
$this->renameModuleAndActionInRequest();
}
/**
@ -134,19 +142,23 @@ class Request
* we rewrite to correct renamed plugin: Referrers
*
* @param $module
* @return string
* @param $action
* @return array( $module, $action )
* @ignore
*/
public static function renameModule($module)
public static function getRenamedModuleAndAction($module, $action)
{
$moduleToRedirect = array(
'Referers' => 'Referrers',
'PDFReports' => 'ScheduledReports',
);
if (isset($moduleToRedirect[$module])) {
return $moduleToRedirect[$module];
}
return $module;
/**
* This event is posted in the Request dispatcher and can be used
* to overwrite the Module and Action to dispatch.
* This is useful when some Controller methods or API methods have been renamed or moved to another plugin.
*
* @param $module string
* @param $action string
*/
Piwik::postEvent('Request.getRenamedModuleAndAction', array(&$module, &$action));
return array($module, $action);
}
/**
@ -168,9 +180,9 @@ class Request
/**
* Dispatches the API request to the appropriate API method and returns the result
* after post-processing.
*
*
* Post-processing includes:
*
*
* - flattening if **flat** is 0
* - running generic filters unless **disable_generic_filters** is set to 1
* - URL decoding label column values
@ -178,10 +190,10 @@ class Request
* - removing columns based on the values of the **hideColumns** and **showColumns** query parameters
* - filtering rows if the **label** query parameter is set
* - converting the result to the appropriate format (ie, XML, JSON, etc.)
*
*
* If `'original'` is supplied for the output format, the result is returned as a PHP
* object.
*
*
* @throws PluginDeactivatedException if the module plugin is not activated.
* @throws Exception if the requested API method cannot be called, if required parameters for the
* API method are missing or if the API method throws an exception and the **format**
@ -196,42 +208,90 @@ class Request
// create the response
$response = new ResponseBuilder($outputFormat, $this->request);
$corsHandler = new CORSHandler();
$corsHandler->handle();
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);
$shouldReloadAuth = false;
try {
// read parameters
$moduleMethod = Common::getRequestVar('method', null, 'string', $this->request);
list($module, $method) = $this->extractModuleAndMethod($moduleMethod);
list($module, $method) = self::getRenamedModuleAndAction($module, $method);
PluginManager::getInstance()->checkIsPluginActivated($module);
$module = $this->renameModule($module);
$apiClassName = self::getClassNameAPI($module);
if (!\Piwik\Plugin\Manager::getInstance()->isPluginActivated($module)) {
throw new PluginDeactivatedException($module);
if ($shouldReloadAuth = self::shouldReloadAuthUsingTokenAuth($this->request)) {
$access = Access::getInstance();
$tokenAuthToRestore = $access->getTokenAuth();
$hadSuperUserAccess = $access->hasSuperUserAccess();
self::forceReloadAuthUsingTokenAuth($tokenAuth);
}
$apiClassName = $this->getClassNameAPI($module);
self::reloadAuthUsingTokenAuth($this->request);
// call the method
$returnedValue = Proxy::getInstance()->call($apiClassName, $method, $this->request);
$toReturn = $response->getResponse($returnedValue, $module, $method);
} catch (Exception $e) {
Log::debug($e);
$toReturn = $response->getResponseException($e);
}
if ($shouldReloadAuth) {
$this->restoreAuthUsingTokenAuth($tokenAuthToRestore, $hadSuperUserAccess);
}
return $toReturn;
}
private function restoreAuthUsingTokenAuth($tokenToRestore, $hadSuperUserAccess)
{
// if we would not make sure to unset super user access, the tokenAuth would be not authenticated and any
// token would just keep super user access (eg if the token that was reloaded before had super user access)
Access::getInstance()->setSuperUserAccess(false);
// we need to restore by reloading the tokenAuth as some permissions could have been removed in the API
// request etc. Otherwise we could just store a clone of Access::getInstance() and restore here
self::forceReloadAuthUsingTokenAuth($tokenToRestore);
if ($hadSuperUserAccess && !Access::getInstance()->hasSuperUserAccess()) {
// we are in context of `doAsSuperUser()` and need to restore this behaviour
Access::getInstance()->setSuperUserAccess(true);
}
}
/**
* Returns the name of a plugin's API class by plugin name.
*
*
* @param string $plugin The plugin name, eg, `'Referrers'`.
* @return string The fully qualified API class name, eg, `'\Piwik\Plugins\Referrers\API'`.
*/
static public function getClassNameAPI($plugin)
public static function getClassNameAPI($plugin)
{
return sprintf('\Piwik\Plugins\%s\API', $plugin);
}
/**
* Detect if request is an API request. Meaning the module is 'API' and an API method having a valid format was
* specified.
*
* @param array $request eg array('module' => 'API', 'method' => 'Test.getMethod')
* @return bool
* @throws Exception
*/
public static function isApiRequest($request)
{
$module = Common::getRequestVar('module', '', 'string', $request);
$method = Common::getRequestVar('method', '', 'string', $request);
return $module === 'API' && !empty($method) && (count(explode('.', $method)) === 2);
}
/**
* If the token_auth is found in the $request parameter,
* the current session will be authenticated using this token_auth.
@ -241,28 +301,60 @@ class Request
* @return void
* @ignore
*/
static public function reloadAuthUsingTokenAuth($request = null)
public static function reloadAuthUsingTokenAuth($request = null)
{
// if a token_auth is specified in the API request, we load the right permissions
$token_auth = Common::getRequestVar('token_auth', '', 'string', $request);
if ($token_auth) {
/**
* Triggered when authenticating an API request, but only if the **token_auth**
* query parameter is found in the request.
*
* Plugins that provide authentication capabilities should subscribe to this event
* and make sure the global authentication object (the object returned by `Registry::get('auth')`)
* is setup to use `$token_auth` when its `authenticate()` method is executed.
*
* @param string $token_auth The value of the **token_auth** query parameter.
*/
Piwik::postEvent('API.Request.authenticate', array($token_auth));
Access::getInstance()->reloadAccess();
SettingsServer::raiseMemoryLimitIfNecessary();
if (self::shouldReloadAuthUsingTokenAuth($request)) {
self::forceReloadAuthUsingTokenAuth($token_auth);
}
}
/**
* The current session will be authenticated using this token_auth.
* It will overwrite the previous Auth object.
*
* @param string $tokenAuth
* @return void
*/
private static function forceReloadAuthUsingTokenAuth($tokenAuth)
{
/**
* Triggered when authenticating an API request, but only if the **token_auth**
* query parameter is found in the request.
*
* Plugins that provide authentication capabilities should subscribe to this event
* and make sure the global authentication object (the object returned by `StaticContainer::get('Piwik\Auth')`)
* is setup to use `$token_auth` when its `authenticate()` method is executed.
*
* @param string $token_auth The value of the **token_auth** query parameter.
*/
Piwik::postEvent('API.Request.authenticate', array($tokenAuth));
Access::getInstance()->reloadAccess();
SettingsServer::raiseMemoryLimitIfNecessary();
}
private static function shouldReloadAuthUsingTokenAuth($request)
{
if (is_null($request)) {
$request = self::getDefaultRequest();
}
if (!isset($request['token_auth'])) {
// no token is given so we just keep the current loaded user
return false;
}
// a token is specified, we need to reload auth in case it is different than the current one, even if it is empty
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $request);
// not using !== is on purpose as getTokenAuth() might return null whereas $tokenAuth is '' . In this case
// we do not need to reload.
return $tokenAuth != Access::getInstance()->getTokenAuth();
}
/**
* Returns array($class, $method) from the given string $class.$method
*
@ -286,18 +378,23 @@ class Request
* @param string $method The API method to call, ie, `'Actions.getPageTitles'`.
* @param array $paramOverride The parameter name-value pairs to use instead of what's
* in `$_GET` & `$_POST`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*
* To avoid using any parameters from $_GET or $_POST, set this to an empty `array()`.
* @return mixed The result of the API request. See {@link process()}.
*/
public static function processRequest($method, $paramOverride = array())
public static function processRequest($method, $paramOverride = array(), $defaultRequest = null)
{
$params = array();
$params['format'] = 'original';
$params['serialize'] = '0';
$params['module'] = 'API';
$params['method'] = $method;
$params = $paramOverride + $params;
// process request
$request = new Request($params);
$request = new Request($params, $defaultRequest);
return $request->process();
}
@ -305,7 +402,7 @@ class Request
* Returns the original request parameters in the current query string as an array mapping
* query parameter names with values. The result of this function will not be affected
* by any modifications to `$_GET` and will not include parameters in `$_POST`.
*
*
* @return array
*/
public static function getRequestParametersGET()
@ -343,7 +440,7 @@ class Request
// unless the filter param was in $queryParams
$genericFiltersInfo = DataTableGenericFilter::getGenericFiltersInformation();
foreach ($genericFiltersInfo as $filter) {
foreach ($filter as $queryParamName => $queryParamInfo) {
foreach ($filter[1] as $queryParamName => $queryParamInfo) {
if (!isset($params[$queryParamName])) {
$params[$queryParamName] = null;
}
@ -379,10 +476,10 @@ class Request
/**
* Returns the segment query parameter from the original request, without modifications.
*
*
* @return array|bool
*/
static public function getRawSegmentFromRequest()
public static function getRawSegmentFromRequest()
{
// we need the URL encoded segment parameter, we fetch it from _SERVER['QUERY_STRING'] instead of default URL decoded _GET
$segmentRaw = false;
@ -395,4 +492,23 @@ class Request
}
return $segmentRaw;
}
private function renameModuleAndActionInRequest()
{
if (empty($this->request['apiModule'])) {
return;
}
if (empty($this->request['apiAction'])) {
$this->request['apiAction'] = null;
}
list($this->request['apiModule'], $this->request['apiAction']) = $this->getRenamedModuleAndAction($this->request['apiModule'], $this->request['apiAction']);
}
/**
* @return array
*/
private static function getDefaultRequest()
{
return $_GET + $_POST;
}
}

506
www/analytics/core/API/ResponseBuilder.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,21 +9,22 @@
namespace Piwik\API;
use Exception;
use Piwik\API\DataTableManipulator\Flattener;
use Piwik\API\DataTableManipulator\LabelFilter;
use Piwik\API\DataTableManipulator\ReportTotalsCalculator;
use Piwik\Common;
use Piwik\DataTable\Renderer\Json;
use Piwik\DataTable\Renderer;
use Piwik\DataTable\Simple;
use Piwik\DataTable;
use Piwik\DataTable\Renderer;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\ColumnDelete;
use Piwik\DataTable\Filter\Pattern;
/**
*/
class ResponseBuilder
{
private $request = null;
private $outputFormat = null;
private $apiRenderer = null;
private $request = null;
private $sendHeader = true;
private $postProcessDataTable = true;
private $apiModule = false;
private $apiMethod = false;
@ -34,8 +35,19 @@ class ResponseBuilder
*/
public function __construct($outputFormat, $request = array())
{
$this->request = $request;
$this->outputFormat = $outputFormat;
$this->request = $request;
$this->apiRenderer = ApiRenderer::factory($outputFormat, $request);
}
public function disableSendHeader()
{
$this->sendHeader = false;
}
public function disableDataTablePostProcessor()
{
$this->postProcessDataTable = false;
}
/**
@ -70,61 +82,21 @@ class ResponseBuilder
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
if($this->outputFormat == 'original') {
@header('Content-Type: text/plain; charset=utf-8');
}
return $this->renderValue($value);
}
$this->sendHeaderIfEnabled();
/**
* Returns an error $message in the requested $format
*
* @param Exception $e
* @throws Exception
* @return string
*/
public function getResponseException(Exception $e)
{
$format = strtolower($this->outputFormat);
if ($format == 'original') {
throw $e;
}
try {
$renderer = Renderer::factory($format);
} catch (Exception $exceptionRenderer) {
return "Error: " . $e->getMessage() . " and: " . $exceptionRenderer->getMessage();
}
$e = $this->decorateExceptionWithDebugTrace($e);
$renderer->setException($e);
if ($format == 'php') {
$renderer->setSerialize($this->caseRendererPHPSerialize());
}
return $renderer->renderException();
}
/**
* @param $value
* @return string
*/
protected function renderValue($value)
{
// when null or void is returned from the api call, we handle it as a successful operation
if (!isset($value)) {
return $this->handleSuccess();
if (ob_get_contents()) {
return null;
}
return $this->apiRenderer->renderSuccess('ok');
}
// If the returned value is an object DataTable we
// apply the set of generic filters if asked in the URL
// and we render the DataTable according to the format specified in the URL
if ($value instanceof DataTable
|| $value instanceof DataTable\Map
) {
if ($value instanceof DataTableInterface) {
return $this->handleDataTable($value);
}
@ -137,26 +109,39 @@ class ResponseBuilder
return $this->handleArray($value);
}
// original data structure requested, we return without process
if ($this->outputFormat == 'original') {
return $value;
if (is_object($value)) {
return $this->apiRenderer->renderObject($value);
}
if (is_object($value)
|| is_resource($value)
) {
return $this->getResponseException(new Exception('The API cannot handle this data structure.'));
if (is_resource($value)) {
return $this->apiRenderer->renderResource($value);
}
// bool // integer // float // serialized object
return $this->handleScalar($value);
return $this->apiRenderer->renderScalar($value);
}
/**
* Returns an error $message in the requested $format
*
* @param Exception $e
* @throws Exception
* @return string
*/
public function getResponseException(Exception $e)
{
$e = $this->decorateExceptionWithDebugTrace($e);
$message = $this->formatExceptionMessage($e);
$this->sendHeaderIfEnabled();
return $this->apiRenderer->renderException($message, $e);
}
/**
* @param Exception $e
* @return Exception
*/
protected function decorateExceptionWithDebugTrace(Exception $e)
private function decorateExceptionWithDebugTrace(Exception $e)
{
// If we are in tests, show full backtrace
if (defined('PIWIK_PATH_TEST_TO_ROOT')) {
@ -165,314 +150,109 @@ class ResponseBuilder
} else {
$message = $e->getMessage() . "\n \n --> To temporarily debug this error further, set const PIWIK_PRINT_ERROR_BACKTRACE=true; in index.php";
}
return new Exception($message);
}
return $e;
}
/**
* Returns true if the user requested to serialize the output data (&serialize=1 in the request)
*
* @param mixed $defaultSerializeValue Default value in case the user hasn't specified a value
* @return bool
*/
protected function caseRendererPHPSerialize($defaultSerializeValue = 1)
private function formatExceptionMessage(Exception $exception)
{
$serialize = Common::getRequestVar('serialize', $defaultSerializeValue, 'int', $this->request);
if ($serialize) {
$message = $exception->getMessage();
if (\Piwik_ShouldPrintBackTraceWithMessage()) {
$message .= "\n" . $exception->getTraceAsString();
}
return Renderer::formatValueXml($message);
}
private function handleDataTable(DataTableInterface $datatable)
{
if ($this->postProcessDataTable) {
$postProcessor = new DataTablePostProcessor($this->apiModule, $this->apiMethod, $this->request);
$datatable = $postProcessor->process($datatable);
}
return $this->apiRenderer->renderDataTable($datatable);
}
private function handleArray($array)
{
$firstArray = null;
$firstKey = null;
if (!empty($array)) {
$firstArray = reset($array);
$firstKey = key($array);
}
$isAssoc = !empty($firstArray) && is_numeric($firstKey) && is_array($firstArray) && count(array_filter(array_keys($firstArray), 'is_string'));
if (is_numeric($firstKey)) {
$columns = Common::getRequestVar('filter_column', false, 'array', $this->request);
$pattern = Common::getRequestVar('filter_pattern', '', 'string', $this->request);
if ($columns != array(false) && $pattern !== '') {
$pattern = new Pattern(new DataTable(), $columns, $pattern);
$array = $pattern->filterArray($array);
}
$limit = Common::getRequestVar('filter_limit', -1, 'integer', $this->request);
$offset = Common::getRequestVar('filter_offset', '0', 'integer', $this->request);
if ($this->shouldApplyLimitOnArray($limit, $offset)) {
$array = array_slice($array, $offset, $limit, $preserveKeys = false);
}
}
if ($isAssoc) {
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
if ($hideColumns !== '' || $showColumns !== '') {
$columnDelete = new ColumnDelete(new DataTable(), $hideColumns, $showColumns);
$array = $columnDelete->filter($array);
}
}
return $this->apiRenderer->renderArray($array);
}
private function shouldApplyLimitOnArray($limit, $offset)
{
if ($limit === -1) {
// all fields are requested
return false;
}
if ($offset > 0) {
// an offset is specified, we have to apply the limit
return true;
}
return false;
// "api_datatable_default_limit" is set by API\Controller if no filter_limit is specified by the user.
// it holds the number of the configured default limit.
$limitSetBySystem = Common::getRequestVar('api_datatable_default_limit', -2, 'integer', $this->request);
// we ignore the limit if the datatable_default_limit was set by the system as this default filter_limit is
// only meant for dataTables but not for arrays. This way we stay BC as filter_limit was not applied pre
// Piwik 2.6 and some fixes were made in Piwik 2.13.
$wasFilterLimitSetBySystem = $limitSetBySystem !== -2;
// we check for "$limitSetBySystem === $limit" as an API method could request another API method with
// another limit. In this case we need to apply it again.
$isLimitStillDefaultLimit = $limitSetBySystem === $limit;
if ($wasFilterLimitSetBySystem && $isLimitStillDefaultLimit) {
return false;
}
return true;
}
/**
* Apply the specified renderer to the DataTable
*
* @param DataTable|array $dataTable
* @return string
*/
protected function getRenderedDataTable($dataTable)
private function sendHeaderIfEnabled()
{
$format = strtolower($this->outputFormat);
// if asked for original dataStructure
if ($format == 'original') {
// by default "original" data is not serialized
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
$dataTable = serialize($dataTable);
}
return $dataTable;
if ($this->sendHeader) {
$this->apiRenderer->sendHeader();
}
$method = Common::getRequestVar('method', '', 'string', $this->request);
$renderer = Renderer::factory($format);
$renderer->setTable($dataTable);
$renderer->setRenderSubTables(Common::getRequestVar('expanded', false, 'int', $this->request));
$renderer->setHideIdSubDatableFromResponse(Common::getRequestVar('hideIdSubDatable', false, 'int', $this->request));
if ($format == 'php') {
$renderer->setSerialize($this->caseRendererPHPSerialize());
$renderer->setPrettyDisplay(Common::getRequestVar('prettyDisplay', false, 'int', $this->request));
} else if ($format == 'html') {
$renderer->setTableId($this->request['method']);
} else if ($format == 'csv' || $format == 'tsv') {
$renderer->setConvertToUnicode(Common::getRequestVar('convertToUnicode', true, 'int', $this->request));
}
// prepare translation of column names
if ($format == 'html' || $format == 'csv' || $format == 'tsv' || $format = 'rss') {
$renderer->setApiMethod($method);
$renderer->setIdSite(Common::getRequestVar('idSite', false, 'int', $this->request));
$renderer->setTranslateColumnNames(Common::getRequestVar('translateColumnNames', false, 'int', $this->request));
}
return $renderer->render();
}
/**
* Returns a success $message in the requested $format
*
* @param string $message
* @return string
*/
protected function handleSuccess($message = 'ok')
{
// return a success message only if no content has already been buffered, useful when APIs return raw text or html content to the browser
if (!ob_get_contents()) {
switch ($this->outputFormat) {
case 'xml':
@header("Content-Type: text/xml;charset=utf-8");
$return =
"<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n" .
"<result>\n" .
"\t<success message=\"" . $message . "\" />\n" .
"</result>";
break;
case 'json':
@header("Content-Type: application/json");
$return = '{"result":"success", "message":"' . $message . '"}';
break;
case 'php':
$return = array('result' => 'success', 'message' => $message);
if ($this->caseRendererPHPSerialize()) {
$return = serialize($return);
}
break;
case 'csv':
@header("Content-Type: application/vnd.ms-excel");
@header("Content-Disposition: attachment; filename=piwik-report-export.csv");
$return = "message\n" . $message;
break;
default:
$return = 'Success:' . $message;
break;
}
return $return;
}
}
/**
* Converts the given scalar to an data table
*
* @param mixed $scalar
* @return string
*/
protected function handleScalar($scalar)
{
$dataTable = new Simple();
$dataTable->addRowsFromArray(array($scalar));
return $this->getRenderedDataTable($dataTable);
}
/**
* Handles the given data table
*
* @param DataTable $datatable
* @return string
*/
protected function handleDataTable($datatable)
{
// if requested, flatten nested tables
if (Common::getRequestVar('flat', '0', 'string', $this->request) == '1') {
$flattener = new Flattener($this->apiModule, $this->apiMethod, $this->request);
if (Common::getRequestVar('include_aggregate_rows', '0', 'string', $this->request) == '1') {
$flattener->includeAggregateRows();
}
$datatable = $flattener->flatten($datatable);
}
if (1 == Common::getRequestVar('totals', '1', 'integer', $this->request)) {
$genericFilter = new ReportTotalsCalculator($this->apiModule, $this->apiMethod, $this->request);
$datatable = $genericFilter->calculate($datatable);
}
// if the flag disable_generic_filters is defined we skip the generic filters
if (0 == Common::getRequestVar('disable_generic_filters', '0', 'string', $this->request)) {
$genericFilter = new DataTableGenericFilter($this->request);
$genericFilter->filter($datatable);
}
// we automatically safe decode all datatable labels (against xss)
$datatable->queueFilter('SafeDecodeLabel');
// if the flag disable_queued_filters is defined we skip the filters that were queued
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $this->request) == 0) {
$datatable->applyQueuedFilters();
}
// use the ColumnDelete filter if hideColumns/showColumns is provided (must be done
// after queued filters are run so processed metrics can be removed, too)
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
if ($hideColumns !== '' || $showColumns !== '') {
$datatable->filter('ColumnDelete', array($hideColumns, $showColumns));
}
// apply label filter: only return rows matching the label parameter (more than one if more than one label)
$label = $this->getLabelFromRequest($this->request);
if (!empty($label)) {
$addLabelIndex = Common::getRequestVar('labelFilterAddLabelIndex', 0, 'int', $this->request) == 1;
$filter = new LabelFilter($this->apiModule, $this->apiMethod, $this->request);
$datatable = $filter->filter($label, $datatable, $addLabelIndex);
}
return $this->getRenderedDataTable($datatable);
}
/**
* Converts the given simple array to a data table
*
* @param array $array
* @return string
*/
protected function handleArray($array)
{
if ($this->outputFormat == 'original') {
// we handle the serialization. Because some php array have a very special structure that
// couldn't be converted with the automatic DataTable->addRowsFromSimpleArray
// the user may want to request the original PHP data structure serialized by the API
// in case he has to setup serialize=1 in the URL
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
return serialize($array);
}
return $array;
}
$multiDimensional = $this->handleMultiDimensionalArray($array);
if ($multiDimensional !== false) {
return $multiDimensional;
}
return $this->getRenderedDataTable($array);
}
/**
* Is this a multi dimensional array?
* Multi dim arrays are not supported by the Datatable renderer.
* We manually render these.
*
* array(
* array(
* 1,
* 2 => array( 1,
* 2
* )
* ),
* array( 2,
* 3
* )
* );
*
* @param array $array
* @return string|bool false if it isn't a multidim array
*/
protected function handleMultiDimensionalArray($array)
{
$first = reset($array);
foreach ($array as $first) {
if (is_array($first)) {
foreach ($first as $key => $value) {
// Yes, this is a multi dim array
if (is_array($value)) {
switch ($this->outputFormat) {
case 'json':
@header("Content-Type: application/json");
return self::convertMultiDimensionalArrayToJson($array);
break;
case 'php':
if ($this->caseRendererPHPSerialize($defaultSerialize = 0)) {
return serialize($array);
}
return $array;
case 'xml':
@header("Content-Type: text/xml;charset=utf-8");
return $this->getRenderedDataTable($array);
default:
break;
}
}
}
}
}
return false;
}
/**
* Render a multidimensional array to Json
* Handle DataTable|Set elements in the first dimension only, following case does not work:
* array(
* array(
* DataTable,
* 2 => array(
* 1,
* 2
* ),
* ),
* );
*
* @param array $array can contain scalar, arrays, DataTable and Set
* @return string
*/
public static function convertMultiDimensionalArrayToJson($array)
{
$jsonRenderer = new Json();
$jsonRenderer->setTable($array);
$renderedReport = $jsonRenderer->render();
return $renderedReport;
}
/**
* Returns the value for the label query parameter which can be either a string
* (ie, label=...) or array (ie, label[]=...).
*
* @param array $request
* @return array
*/
static public function getLabelFromRequest($request)
{
$label = Common::getRequestVar('label', array(), 'array', $request);
if (empty($label)) {
$label = Common::getRequestVar('label', '', 'string', $request);
if (!empty($label)) {
$label = array($label);
}
}
$label = self::unsanitizeLabelParameter($label);
return $label;
}
static public function unsanitizeLabelParameter($label)
{
// this is needed because Proxy uses Common::getRequestVar which in turn
// uses Common::sanitizeInputValue. This causes the > that separates recursive labels
// to become &gt; and we need to undo that here.
$label = Common::unsanitizeInputValues($label);
return $label;
}
}

212
www/analytics/core/Access.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,55 +8,32 @@
*/
namespace Piwik;
use Piwik\Db;
use Piwik\Plugins\UsersManager\API as APIUsersManager;
use Exception;
use Piwik\Container\StaticContainer;
/**
* Singleton that manages user access to Piwik resources.
*
*
* To check whether a user has access to a resource, use one of the {@link Piwik Piwik::checkUser...}
* methods.
*
* In Piwik there are four different access levels:
*
*
* - **no access**: Users with this access level cannot view the resource.
* - **view access**: Users with this access level can view the resource, but cannot modify it.
* - **admin access**: Users with this access level can view and modify the resource.
* - **Super User access**: Only the Super User has this access level. It means the user can do
* whatever he/she wants.
*
*
* Super user access is required to set some configuration options.
* All other options are specific to the user or to a website.
*
* Access is granted per website. Uses with access for a website can view all
* data associated with that website.
*
*
*/
class Access
{
private static $instance = null;
/**
* Gets the singleton instance. Creates it if necessary.
*/
public static function getInstance()
{
if (self::$instance == null) {
self::$instance = new self;
Piwik::postEvent('Access.createAccessSingleton', array(&self::$instance));
}
return self::$instance;
}
/**
* Sets the singleton instance. For testing purposes.
*/
public static function setSingletonInstance($instance)
{
self::$instance = $instance;
}
/**
* Array of idsites available to the current user, indexed by permission level
* @see getSitesIdWith*()
@ -101,6 +78,16 @@ class Access
*/
private $auth = null;
/**
* Gets the singleton instance. Creates it if necessary.
*
* @return self
*/
public static function getInstance()
{
return StaticContainer::get('Piwik\Access');
}
/**
* Returns the list of the existing Access level.
* Useful when a given API method requests a given acccess Level.
@ -117,6 +104,11 @@ class Access
* Constructor
*/
public function __construct()
{
$this->resetSites();
}
private function resetSites()
{
$this->idsitesByAccess = array(
'view' => array(),
@ -138,15 +130,22 @@ class Access
*/
public function reloadAccess(Auth $auth = null)
{
if (!is_null($auth)) {
$this->resetSites();
if (isset($auth)) {
$this->auth = $auth;
}
// if the Auth wasn't set, we may be in the special case of setSuperUser(), otherwise we fail
if (is_null($this->auth)) {
if ($this->hasSuperUserAccess()) {
return $this->reloadAccessSuperUser();
}
if ($this->hasSuperUserAccess()) {
$this->makeSureLoginNameIsSet();
return true;
}
$this->token_auth = null;
$this->login = null;
// if the Auth wasn't set, we may be in the special case of setSuperUser(), otherwise we fail TODO: docs + review
if ($this->auth === null) {
return false;
}
@ -156,28 +155,23 @@ class Access
if (!$result->wasAuthenticationSuccessful()) {
return false;
}
$this->login = $result->getIdentity();
$this->token_auth = $result->getTokenAuth();
// case the superUser is logged in
if ($result->hasSuperUserAccess()) {
return $this->reloadAccessSuperUser();
$this->setSuperUserAccess(true);
}
// in case multiple calls to API using different tokens, we ensure we reset it as not SU
$this->setSuperUserAccess(false);
// we join with site in case there are rows in access for an idsite that doesn't exist anymore
// (backward compatibility ; before we deleted the site without deleting rows in _access table)
$accessRaw = $this->getRawSitesWithSomeViewAccess($this->login);
foreach ($accessRaw as $access) {
$this->idsitesByAccess[$access['access']][] = $access['idsite'];
}
return true;
}
public function getRawSitesWithSomeViewAccess($login)
{
return Db::fetchAll(self::getSqlAccessSite("access, t2.idsite"), $login);
$sql = self::getSqlAccessSite("access, t2.idsite");
return Db::fetchAll($sql, $login);
}
/**
@ -188,29 +182,50 @@ class Access
*/
public static function getSqlAccessSite($select)
{
return "SELECT " . $select . "
FROM " . Common::prefixTable('access') . " as t1
JOIN " . Common::prefixTable('site') . " as t2 USING (idsite) " .
" WHERE login = ?";
$access = Common::prefixTable('access');
$siteTable = Common::prefixTable('site');
return "SELECT " . $select . " FROM " . $access . " as t1
JOIN " . $siteTable . " as t2 USING (idsite) WHERE login = ?";
}
/**
* Reload Super User access
* Make sure a login name is set
*
* @return bool
* @return true
*/
protected function reloadAccessSuperUser()
protected function makeSureLoginNameIsSet()
{
$this->hasSuperUserAccess = true;
try {
$allSitesId = Plugins\SitesManager\API::getInstance()->getAllSitesId();
} catch (\Exception $e) {
$allSitesId = array();
if (empty($this->login)) {
// flag to force non empty login so Super User is not mistaken for anonymous
$this->login = 'super user was set';
}
$this->idsitesByAccess['superuser'] = $allSitesId;
}
return true;
protected function loadSitesIfNeeded()
{
if ($this->hasSuperUserAccess) {
if (empty($this->idsitesByAccess['superuser'])) {
try {
$allSitesId = Plugins\SitesManager\API::getInstance()->getAllSitesId();
} catch (\Exception $e) {
$allSitesId = array();
}
$this->idsitesByAccess['superuser'] = $allSitesId;
}
} elseif (isset($this->login)) {
if (empty($this->idsitesByAccess['view'])
&& empty($this->idsitesByAccess['admin'])) {
// we join with site in case there are rows in access for an idsite that doesn't exist anymore
// (backward compatibility ; before we deleted the site without deleting rows in _access table)
$accessRaw = $this->getRawSitesWithSomeViewAccess($this->login);
foreach ($accessRaw as $access) {
$this->idsitesByAccess[$access['access']][] = $access['idsite'];
}
}
}
}
/**
@ -221,12 +236,12 @@ class Access
*/
public function setSuperUserAccess($bool = true)
{
if ($bool) {
$this->reloadAccessSuperUser();
} else {
$this->hasSuperUserAccess = false;
$this->idsitesByAccess['superuser'] = array();
$this->hasSuperUserAccess = (bool) $bool;
if ($bool) {
$this->makeSureLoginNameIsSet();
} else {
$this->resetSites();
}
}
@ -269,6 +284,8 @@ class Access
*/
public function getSitesIdWithAtLeastViewAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['view'],
$this->idsitesByAccess['admin'],
@ -284,13 +301,14 @@ class Access
*/
public function getSitesIdWithAdminAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['admin'],
$this->idsitesByAccess['superuser'])
);
}
/**
* Returns an array of ID sites for which the user has a VIEW access only.
*
@ -300,6 +318,8 @@ class Access
*/
public function getSitesIdWithViewAccess()
{
$this->loadSitesIfNeeded();
return $this->idsitesByAccess['view'];
}
@ -315,6 +335,22 @@ class Access
}
}
/**
* Returns `true` if the current user has admin access to at least one site.
*
* @return bool
*/
public function isUserHasSomeAdminAccess()
{
if ($this->hasSuperUserAccess()) {
return true;
}
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
return count($idSitesAccessible) > 0;
}
/**
* If the user doesn't have an ADMIN access for at least one website, throws an exception
*
@ -322,11 +358,7 @@ class Access
*/
public function checkUserHasSomeAdminAccess()
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
if (count($idSitesAccessible) == 0) {
if (!$this->isUserHasSomeAdminAccess()) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('admin')));
}
}
@ -341,7 +373,9 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
if (count($idSitesAccessible) == 0) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('view')));
}
@ -359,8 +393,10 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'admin'", $idsite)));
@ -380,8 +416,10 @@ class Access
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
throw new NoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'view'", $idsite)));
@ -401,11 +439,41 @@ class Access
}
$idSites = Site::getIdSitesFromIdSitesString($idSites);
if (empty($idSites)) {
throw new NoAccessException("The parameter 'idSite=' is missing from the request.");
}
return $idSites;
}
/**
* Executes a callback with superuser privileges, making sure those privileges are rescinded
* before this method exits. Privileges will be rescinded even if an exception is thrown.
*
* @param callback $function The callback to execute. Should accept no arguments.
* @return mixed The result of `$function`.
* @throws Exception rethrows any exceptions thrown by `$function`.
* @api
*/
public static function doAsSuperUser($function)
{
$isSuperUser = self::getInstance()->hasSuperUserAccess();
self::getInstance()->setSuperUserAccess(true);
try {
$result = $function();
} catch (Exception $ex) {
self::getInstance()->setSuperUserAccess($isSuperUser);
throw $ex;
}
self::getInstance()->setSuperUserAccess($isSuperUser);
return $result;
}
}
/**

246
www/analytics/core/Application/Environment.php Normal file
View file

@ -0,0 +1,246 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use DI\Container;
use Piwik\Application\Kernel\EnvironmentValidator;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
use Piwik\Container\ContainerFactory;
use Piwik\Container\StaticContainer;
use Piwik\Piwik;
/**
* Encapsulates Piwik environment setup and access.
*
* The Piwik environment consists of two main parts: the kernel and the DI container.
*
* The 'kernel' is the core part of Piwik that cannot be modified / extended through the DI container.
* It includes components that are required to create the DI container.
*
* Currently the only objects in the 'kernel' are a GlobalSettingsProvider object and a
* PluginList object. The GlobalSettingsProvider object is required for the current PluginList
* implementation and for checking whether Development mode is enabled. The PluginList is
* needed in order to determine what plugins are activated, since plugins can provide their
* own DI configuration.
*
* The DI container contains every other Piwik object, including the Plugin\Manager,
* plugin API instances, dependent services, etc. Plugins and users can override/extend
* the objects in this container.
*
* NOTE: DI support in Piwik is currently a work in process; not everything is currently
* stored in the DI container, but we are working towards this.
*/
class Environment
{
/**
* @internal
* @var EnvironmentManipulator
*/
private static $globalEnvironmentManipulator = null;
/**
* @var string
*/
private $environment;
/**
* @var array
*/
private $definitions;
/**
* @var Container
*/
private $container;
/**
* @var GlobalSettingsProvider
*/
private $globalSettingsProvider;
/**
* @var PluginList
*/
private $pluginList;
/**
* @param string $environment
* @param array $definitions
*/
public function __construct($environment, array $definitions = array())
{
$this->environment = $environment;
$this->definitions = $definitions;
}
/**
* Initializes the kernel globals and DI container.
*/
public function init()
{
$this->invokeBeforeContainerCreatedHook();
$this->container = $this->createContainer();
StaticContainer::push($this->container);
$this->validateEnvironment();
$this->invokeEnvironmentBootstrappedHook();
Piwik::postEvent('Environment.bootstrapped'); // this event should be removed eventually
}
/**
* Destroys an environment. MUST be called when embedding environments.
*/
public function destroy()
{
StaticContainer::pop();
}
/**
* Returns the DI container. All Piwik objects for a specific Piwik instance should be stored
* in this container.
*
* @return Container
*/
public function getContainer()
{
return $this->container;
}
/**
* @link http://php-di.org/doc/container-configuration.html
*/
private function createContainer()
{
$pluginList = $this->getPluginListCached();
$settings = $this->getGlobalSettingsCached();
$extraDefinitions = $this->getExtraDefinitionsFromManipulators();
$definitions = array_merge(StaticContainer::getDefinitions(), $extraDefinitions, array($this->definitions));
$environments = array($this->environment);
$environments = array_merge($environments, $this->getExtraEnvironmentsFromManipulators());
$containerFactory = new ContainerFactory($pluginList, $settings, $environments, $definitions);
return $containerFactory->create();
}
protected function getGlobalSettingsCached()
{
if ($this->globalSettingsProvider === null) {
$original = $this->getGlobalSettings();
$globalSettingsProvider = $this->getGlobalSettingsProviderOverride($original);
$this->globalSettingsProvider = $globalSettingsProvider ?: $original;
}
return $this->globalSettingsProvider;
}
protected function getPluginListCached()
{
if ($this->pluginList === null) {
$pluginList = $this->getPluginListOverride();
$this->pluginList = $pluginList ?: $this->getPluginList();
}
return $this->pluginList;
}
/**
* Returns the kernel global GlobalSettingsProvider object. Derived classes can override this method
* to provide a different implementation.
*
* @return null|GlobalSettingsProvider
*/
protected function getGlobalSettings()
{
return new GlobalSettingsProvider();
}
/**
* Returns the kernel global PluginList object. Derived classes can override this method to
* provide a different implementation.
*
* @return PluginList
*/
protected function getPluginList()
{
// TODO: in tracker should only load tracker plugins. can't do properly until tracker entrypoint is encapsulated.
return new PluginList($this->getGlobalSettingsCached());
}
private function validateEnvironment()
{
/** @var EnvironmentValidator $validator */
$validator = $this->container->get('Piwik\Application\Kernel\EnvironmentValidator');
$validator->validate();
}
/**
* @param EnvironmentManipulator $manipulator
* @internal
*/
public static function setGlobalEnvironmentManipulator(EnvironmentManipulator $manipulator)
{
self::$globalEnvironmentManipulator = $manipulator;
}
private function getGlobalSettingsProviderOverride(GlobalSettingsProvider $original)
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makeGlobalSettingsProvider($original);
} else {
return null;
}
}
private function invokeBeforeContainerCreatedHook()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->beforeContainerCreated();
}
}
private function getExtraDefinitionsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraDefinitions();
} else {
return array();
}
}
private function invokeEnvironmentBootstrappedHook()
{
if (self::$globalEnvironmentManipulator) {
self::$globalEnvironmentManipulator->onEnvironmentBootstrapped();
}
}
private function getExtraEnvironmentsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraEnvironments();
} else {
return array();
}
}
private function getPluginListOverride()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makePluginList($this->getGlobalSettingsCached());
} else {
return null;
}
}
}

59
www/analytics/core/Application/EnvironmentManipulator.php Normal file
View file

@ -0,0 +1,59 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
/**
* Used to manipulate Environment instances before the container is created.
* Only used by the testing environment setup code, shouldn't be used anywhere
* else.
*/
interface EnvironmentManipulator
{
/**
* Create a custom GlobalSettingsProvider kernel object, overriding the default behavior.
*
* @return GlobalSettingsProvider
*/
public function makeGlobalSettingsProvider(GlobalSettingsProvider $original);
/**
* Create a custom PluginList kernel object, overriding the default behavior.@deprecated
*
* @param GlobalSettingsProvider $globalSettingsProvider
* @return PluginList
*/
public function makePluginList(GlobalSettingsProvider $globalSettingsProvider);
/**
* Invoked before the container is created.
*/
public function beforeContainerCreated();
/**
* Return an array of definition arrays that override DI config specified in PHP config files.
*
* @return array[]
*/
public function getExtraDefinitions();
/**
* Invoked after the container is created and the environment is considered bootstrapped.
*/
public function onEnvironmentBootstrapped();
/**
* Return an array of environment names to apply after the normal environment.
*
* @return string[]
*/
public function getExtraEnvironments();
}

79
www/analytics/core/Application/Kernel/EnvironmentValidator.php Normal file
View file

@ -0,0 +1,79 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Common;
use Piwik\Piwik;
use Piwik\SettingsServer;
use Piwik\Translation\Translator;
/**
* Validates the Piwik environment. This includes making sure the required config files
* are present, and triggering the correct behaviour if otherwise.
*/
class EnvironmentValidator
{
/**
* @var GlobalSettingsProvider
*/
protected $settingsProvider;
/**
* @var Translator
*/
protected $translator;
public function __construct(GlobalSettingsProvider $settingsProvider, Translator $translator)
{
$this->settingsProvider = $settingsProvider;
$this->translator = $translator;
}
public function validate()
{
$inTrackerRequest = SettingsServer::isTrackerApiRequest();
$inConsole = Common::isPhpCliMode();
$this->checkConfigFileExists($this->settingsProvider->getPathGlobal());
$this->checkConfigFileExists($this->settingsProvider->getPathLocal(), $startInstaller = !$inTrackerRequest && !$inConsole);
}
/**
* @param $path
* @param bool $startInstaller
* @throws \Exception
*/
private function checkConfigFileExists($path, $startInstaller = false)
{
if (is_readable($path)) {
return;
}
$message = $this->translator->translate('General_ExceptionConfigurationFileNotFound', array($path));
if (Common::isPhpCliMode()) {
$message .= "\n" . $this->translator->translate('General_ExceptionConfigurationFileNotFound2', array($path, get_current_user()));
}
$exception = new \Exception($message);
if ($startInstaller) {
/**
* Triggered when the configuration file cannot be found or read, which usually
* means Piwik is not installed yet.
*
* This event can be used to start the installation process or to display a custom error message.
*
* @param \Exception $exception The exception that was thrown by `Config::getInstance()`.
*/
Piwik::postEvent('Config.NoConfigurationFile', array($exception), $pending = true);
} else {
throw $exception;
}
}
}

111
www/analytics/core/Application/Kernel/GlobalSettingsProvider.php Normal file
View file

@ -0,0 +1,111 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Config;
use Piwik\Config\IniFileChain;
/**
* Provides global settings. Global settings are organized in sections where
* each section contains a list of name => value pairs. Setting values can
* be primitive values or arrays of primitive values.
*
* Uses the config.ini.php, common.ini.php and global.ini.php files to provide global settings.
*
* At the moment a singleton instance of this class is used in order to get tests to pass.
*/
class GlobalSettingsProvider
{
/**
* @var IniFileChain
*/
protected $iniFileChain;
/**
* @var string
*/
protected $pathGlobal = null;
/**
* @var string
*/
protected $pathCommon = null;
/**
* @var string
*/
protected $pathLocal = null;
/**
* @param string|null $pathGlobal Path to the global.ini.php file. Or null to use the default.
* @param string|null $pathLocal Path to the config.ini.php file. Or null to use the default.
* @param string|null $pathCommon Path to the common.ini.php file. Or null to use the default.
*/
public function __construct($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: Config::getGlobalConfigPath();
$this->pathCommon = $pathCommon ?: Config::getCommonConfigPath();
$this->pathLocal = $pathLocal ?: Config::getLocalConfigPath();
$this->iniFileChain = new IniFileChain();
$this->reload();
}
public function reload($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: $this->pathGlobal;
$this->pathCommon = $pathCommon ?: $this->pathCommon;
$this->pathLocal = $pathLocal ?: $this->pathLocal;
$this->iniFileChain->reload(array($this->pathGlobal, $this->pathCommon), $this->pathLocal);
}
/**
* Returns a settings section.
*
* @param string $name
* @return array
*/
public function &getSection($name)
{
$section =& $this->iniFileChain->get($name);
return $section;
}
/**
* Sets a settings section.
*
* @param string $name
* @param array $value
*/
public function setSection($name, $value)
{
$this->iniFileChain->set($name, $value);
}
public function getIniFileChain()
{
return $this->iniFileChain;
}
public function getPathGlobal()
{
return $this->pathGlobal;
}
public function getPathLocal()
{
return $this->pathLocal;
}
public function getPathCommon()
{
return $this->pathCommon;
}
}

120
www/analytics/core/Application/Kernel/PluginList.php Normal file
View file

@ -0,0 +1,120 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
/**
* Lists the currently activated plugins. Used when setting up Piwik's environment before
* initializing the DI container.
*
* Uses the [Plugins] section in Piwik's INI config to get the activated plugins.
*
* Depends on GlobalSettingsProvider being used.
*
* TODO: parts of Plugin\Manager edit the plugin list; maybe PluginList implementations should be mutable?
*/
class PluginList
{
/**
* @var GlobalSettingsProvider
*/
private $settings;
/**
* Plugins bundled with core package, disabled by default
* @var array
*/
private $corePluginsDisabledByDefault = array(
'DBStats',
'ExampleCommand',
'ExampleSettingsPlugin',
'ExampleUI',
'ExampleVisualization',
'ExamplePluginTemplate',
'ExampleTracker',
'ExampleReport',
'MobileAppMeasurable',
'Provider'
);
// Themes bundled with core package, disabled by default
private $coreThemesDisabledByDefault = array(
'ExampleTheme'
);
public function __construct(GlobalSettingsProvider $settings)
{
$this->settings = $settings;
}
/**
* Returns the list of plugins that should be loaded. Used by the container factory to
* load plugin specific DI overrides.
*
* @return string[]
*/
public function getActivatedPlugins()
{
$section = $this->settings->getSection('Plugins');
return @$section['Plugins'] ?: array();
}
/**
* Returns the list of plugins that are bundled with Piwik.
*
* @return string[]
*/
public function getPluginsBundledWithPiwik()
{
$pathGlobal = $this->settings->getPathGlobal();
$section = $this->settings->getIniFileChain()->getFrom($pathGlobal, 'Plugins');
return $section['Plugins'];
}
/**
* Returns the plugins bundled with core package that are disabled by default.
*
* @return string[]
*/
public function getCorePluginsDisabledByDefault()
{
return array_merge($this->corePluginsDisabledByDefault, $this->coreThemesDisabledByDefault);
}
/**
* Sorts an array of plugins in the order they should be loaded.
*
* @params string[] $plugins
* @return \string[]
*/
public function sortPlugins(array $plugins)
{
$global = $this->getPluginsBundledWithPiwik();
if (empty($global)) {
return $plugins;
}
// we need to make sure a possibly disabled plugin will be still loaded before any 3rd party plugin
$global = array_merge($global, $this->corePluginsDisabledByDefault);
$global = array_values($global);
$plugins = array_values($plugins);
$defaultPluginsLoadedFirst = array_intersect($global, $plugins);
$otherPluginsToLoadAfterDefaultPlugins = array_diff($plugins, $defaultPluginsLoadedFirst);
// sort by name to have a predictable order for those extra plugins
sort($otherPluginsToLoadAfterDefaultPlugins);
$sorted = array_merge($defaultPluginsLoadedFirst, $otherPluginsToLoadAfterDefaultPlugins);
return $sorted;
}
}

421
www/analytics/core/Archive.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,57 +9,58 @@
namespace Piwik;
use Piwik\Archive\Parameters;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Archive\ArchiveInvalidator;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\Period\Range;
use Piwik\Period\Factory as PeriodFactory;
/**
* The **Archive** class is used to query cached analytics statistics
* (termed "archive data").
*
*
* You can use **Archive** instances to get data that was archived for one or more sites,
* for one or more periods and one optional segment.
*
*
* If archive data is not found, this class will initiate the archiving process. [1](#footnote-1)
*
*
* **Archive** instances must be created using the {@link build()} factory method;
* they cannot be constructed.
*
*
* You can search for metrics (such as `nb_visits`) using the {@link getNumeric()} and
* {@link getDataTableFromNumeric()} methods. You can search for
* reports using the {@link getBlob()}, {@link getDataTable()} and {@link getDataTableExpanded()} methods.
*
*
* If you're creating an API that returns report data, you may want to use the
* {@link getDataTableFromArchive()} helper function.
*
*
* ### Learn more
*
*
* Learn more about _archiving_ [here](/guides/all-about-analytics-data).
*
*
* ### Limitations
*
*
* - You cannot get data for multiple range periods in a single query.
* - You cannot get data for periods of different types in a single query.
*
*
* ### Examples
*
*
* **_Querying metrics for an API method_**
*
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* // all sites and multiple dates
* $archive = Archive::build($idSite = 'all', $period = 'month', $date = '2013-01-02,2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* **_Querying and using metrics immediately_**
*
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $data = $archive->getNumeric(array('nb_visits', 'nb_actions'));
*
*
* $visits = $data['nb_visits'];
* $actions = $data['nb_actions'];
*
@ -68,41 +69,40 @@ use Piwik\Period\Range;
* // multiple sites and multiple dates
* $archive = Archive::build($idSite = '1,2,3', $period = 'month', $date = '2013-01-02,2013-03-08');
* $data = $archive->getNumeric('nb_visits');
*
*
* $janSite1Visits = $data['1']['2013-01-01,2013-01-31']['nb_visits'];
* $febSite1Visits = $data['1']['2013-02-01,2013-02-28']['nb_visits'];
* // ... etc.
*
*
* **_Querying for reports_**
*
*
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $dataTable = $archive->getDataTable('MyPlugin_MyReport');
* // ... manipulate $dataTable ...
* return $dataTable;
*
*
* **_Querying a report for an API method_**
*
*
* public function getMyReport($idSite, $period, $date, $segment = false, $expanded = false)
* {
* $dataTable = Archive::getDataTableFromArchive('MyPlugin_MyReport', $idSite, $period, $date, $segment, $expanded);
* $dataTable->queueFilter('ReplaceColumnNames');
* return $dataTable;
* }
*
*
* **_Querying data for multiple range periods_**
*
*
* // get data for first range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-08,2013-03-12');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* // get data for second range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-15,2013-03-20');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
*
* <a name="footnote-1"></a>
* [1]: The archiving process will not be launched if browser archiving is disabled
* and the current request came from a browser (and not the **archive.php** cron
* script).
* and the current request came from a browser.
*
*
* @api
@ -162,6 +162,16 @@ class Archive
*/
private $params;
/**
* @var \Piwik\Cache\Cache
*/
private static $cache;
/**
* @var ArchiveInvalidator
*/
private $invalidator;
/**
* @param Parameters $params
* @param bool $forceIndexedBySite Whether to force index the result of a query by site ID.
@ -173,6 +183,8 @@ class Archive
$this->params = $params;
$this->forceIndexedBySite = $forceIndexedBySite;
$this->forceIndexedByDate = $forceIndexedByDate;
$this->invalidator = StaticContainer::get('Piwik\Archive\ArchiveInvalidator');
}
/**
@ -192,37 +204,42 @@ class Archive
* or date range (ie, 'YYYY-MM-DD,YYYY-MM-DD').
* @param bool|false|string $segment Segment definition or false if no segment should be used. {@link Piwik\Segment}
* @param bool|false|string $_restrictSitesToLogin Used only when running as a scheduled task.
* @param bool $skipAggregationOfSubTables Whether the archive, when it is processed, should also aggregate all sub-tables
* @return Archive
* @return static
*/
public static function build($idSites, $period, $strDate, $segment = false, $_restrictSitesToLogin = false, $skipAggregationOfSubTables = false)
public static function build($idSites, $period, $strDate, $segment = false, $_restrictSitesToLogin = false)
{
$websiteIds = Site::getIdSitesFromIdSitesString($idSites, $_restrictSitesToLogin);
$timezone = false;
if (count($websiteIds) == 1) {
$timezone = Site::getTimezoneFor($websiteIds[0]);
}
if (Period::isMultiplePeriod($strDate, $period)) {
$oPeriod = new Range($period, $strDate);
$oPeriod = PeriodFactory::build($period, $strDate, $timezone);
$allPeriods = $oPeriod->getSubperiods();
} else {
$timezone = count($websiteIds) == 1 ? Site::getTimezoneFor($websiteIds[0]) : false;
$oPeriod = Period::makePeriodFromQueryParams($timezone, $period, $strDate);
$oPeriod = PeriodFactory::makePeriodFromQueryParams($timezone, $period, $strDate);
$allPeriods = array($oPeriod);
}
$segment = new Segment($segment, $websiteIds);
$idSiteIsAll = $idSites == self::REQUEST_ALL_WEBSITES_FLAG;
$segment = new Segment($segment, $websiteIds);
$idSiteIsAll = $idSites == self::REQUEST_ALL_WEBSITES_FLAG;
$isMultipleDate = Period::isMultiplePeriod($strDate, $period);
return Archive::factory($segment, $allPeriods, $websiteIds, $idSiteIsAll, $isMultipleDate, $skipAggregationOfSubTables);
return static::factory($segment, $allPeriods, $websiteIds, $idSiteIsAll, $isMultipleDate);
}
/**
* Returns a new Archive instance that will query archive data for the given set of
* sites and periods, using an optional segment.
*
*
* This method uses an array of Period instances and a Segment instance, instead of strings
* like {@link build()}.
*
*
* If you want to create an Archive instance using data found in query parameters,
* use {@link build()}.
*
*
* @param Segment $segment The segment to use. For no segment, use `new Segment('', $idSites)`.
* @param array $periods An array of Period instances.
* @param array $idSites An array of site IDs (eg, `array(1, 2, 3)`).
@ -232,41 +249,42 @@ class Archive
* @param bool $isMultipleDate Whether multiple dates are being queried or not. If true, then
* the result of querying functions will be indexed by period,
* regardless of whether `count($periods) == 1`.
* @param bool $skipAggregationOfSubTables Whether the archive should skip aggregation of all sub-tables
*
* @return Archive
*/
public static function factory(Segment $segment, array $periods, array $idSites, $idSiteIsAll = false, $isMultipleDate = false, $skipAggregationOfSubTables = false)
public static function factory(Segment $segment, array $periods, array $idSites, $idSiteIsAll = false, $isMultipleDate = false)
{
$forceIndexedBySite = false;
$forceIndexedByDate = false;
if ($idSiteIsAll || count($idSites) > 1) {
$forceIndexedBySite = true;
}
if (count($periods) > 1 || $isMultipleDate) {
$forceIndexedByDate = true;
}
$params = new Parameters($idSites, $periods, $segment, $skipAggregationOfSubTables);
$params = new Parameters($idSites, $periods, $segment);
return new Archive($params, $forceIndexedBySite, $forceIndexedByDate);
return new static($params, $forceIndexedBySite, $forceIndexedByDate);
}
/**
* Queries and returns metric data in an array.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be indexed by site ID first, then period.
*
*
* @param string|array $names One or more archive names, eg, `'nb_visits'`, `'Referrers_distinctKeywords'`,
* etc.
* @return false|numeric|array `false` if there is no data to return, a single numeric value if we're not querying
* @return false|integer|array `false` if there is no data to return, a single numeric value if we're not querying
* for multiple sites/periods, or an array if multiple sites, periods or names are
* queried for.
*/
@ -288,50 +306,22 @@ class Archive
return $result;
}
/**
* Queries and returns blob data in an array.
*
* Reports are stored in blobs as serialized arrays of {@link DataTable\Row} instances, but this
* data can technically be anything. In other words, you can store whatever you want
* as archive data blobs.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be indexed by site ID first, then period.
*
* @param string|array $names One or more archive names, eg, `'Referrers_keywordBySearchEngine'`.
* @param null|string $idSubtable If we're returning serialized DataTable data, then this refers
* to the subtable ID to return. If set to 'all', all subtables
* of each requested report are returned.
* @return array An array of appropriately indexed blob data.
*/
public function getBlob($names, $idSubtable = null)
{
$data = $this->get($names, 'blob', $idSubtable);
return $data->getIndexedArray($this->getResultIndices());
}
/**
* Queries and returns metric data in a DataTable instance.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
*
* _Note: Every DataTable instance returned will have at most one row in it. The contents of each
* row will be the requested metrics for the appropriate site and period._
*
*
* @param string|array $names One or more archive names, eg, 'nb_visits', 'Referrers_distinctKeywords',
* etc.
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
@ -343,22 +333,40 @@ class Archive
return $data->getDataTable($this->getResultIndices());
}
/**
* Similar to {@link getDataTableFromNumeric()} but merges all children on the created DataTable.
*
* This is the same as doing `$this->getDataTableFromNumeric()->mergeChildren()` but this way it is much faster.
*
* @return DataTable|DataTable\Map
*
* @internal Currently only used by MultiSites.getAll plugin. Feel free to remove internal tag if needed somewhere
* else. If no longer needed by MultiSites.getAll please remove this method. If you need this to work in
* a bit different way feel free to refactor as always.
*/
public function getDataTableFromNumericAndMergeChildren($names)
{
$data = $this->get($names, 'numeric');
$resultIndexes = $this->getResultIndices();
return $data->getMergedDataTable($resultIndexes);
}
/**
* Queries and returns one or more reports as DataTable instances.
*
*
* This method will query blob data that is a serialized array of of {@link DataTable\Row}'s and
* unserialize it.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
*
* @param string $name The name of the record to get. This method can only query one record at a time.
* @param int|string|null $idSubtable The ID of the subtable to get (if any).
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
@ -372,13 +380,13 @@ class Archive
/**
* Queries and returns one report with all of its subtables loaded.
*
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map indexed} by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
@ -400,16 +408,18 @@ class Archive
/**
* Returns the list of plugins that archive the given reports.
*
*
* @param array $archiveNames
* @return array
*/
private function getRequestedPlugins($archiveNames)
{
$result = array();
foreach ($archiveNames as $name) {
$result[] = self::getPluginForReport($name);
}
return array_unique($result);
}
@ -427,7 +437,7 @@ class Archive
/**
* Helper function that creates an Archive instance and queries for report data using
* query parameter data. API methods can use this method to reduce code redundancy.
*
*
* @param string $name The name of the report to return.
* @param int|string|array $idSite @see {@link build()}
* @param string $period @see {@link build()}
@ -435,21 +445,19 @@ class Archive
* @param string $segment @see {@link build()}
* @param bool $expanded If true, loads all subtables. See {@link getDataTableExpanded()}
* @param int|null $idSubtable See {@link getDataTableExpanded()}
* @param bool $skipAggregationOfSubTables Whether or not we should skip the aggregation of all sub-tables and only aggregate parent DataTable.
* @param int|null $depth See {@link getDataTableExpanded()}
* @throws \Exception
* @return DataTable|DataTable\Map See {@link getDataTable()} and
* {@link getDataTableExpanded()} for more
* information
* @deprecated Since Piwik 2.12.0 Use Archive::createDataTableFromArchive() instead
*/
public static function getDataTableFromArchive($name, $idSite, $period, $date, $segment, $expanded,
$idSubtable = null, $skipAggregationOfSubTables = false, $depth = null)
$idSubtable = null, $depth = null)
{
Piwik::checkUserHasViewAccess($idSite);
if($skipAggregationOfSubTables && ($expanded || $idSubtable)) {
throw new \Exception("Not expected to skipAggregationOfSubTables when expanded=1 or idSubtable is set.");
}
$archive = Archive::build($idSite, $period, $date, $segment, $_restrictSitesToLogin = false, $skipAggregationOfSubTables);
$archive = Archive::build($idSite, $period, $date, $segment, $_restrictSitesToLogin = false);
if ($idSubtable === false) {
$idSubtable = null;
}
@ -465,9 +473,102 @@ class Archive
return $dataTable;
}
private function appendIdSubtable($recordName, $id)
/**
* Helper function that creates an Archive instance and queries for report data using
* query parameter data. API methods can use this method to reduce code redundancy.
*
* @param string $recordName The name of the report to return.
* @param int|string|array $idSite @see {@link build()}
* @param string $period @see {@link build()}
* @param string $date @see {@link build()}
* @param string $segment @see {@link build()}
* @param bool $expanded If true, loads all subtables. See {@link getDataTableExpanded()}
* @param bool $flat If true, loads all subtables and disabled all recursive filters.
* @param int|null $idSubtable See {@link getDataTableExpanded()}
* @param int|null $depth See {@link getDataTableExpanded()}
* @return DataTable|DataTable\Map
*/
public static function createDataTableFromArchive($recordName, $idSite, $period, $date, $segment, $expanded = false, $flat = false, $idSubtable = null, $depth = null)
{
return $recordName . "_" . $id;
if ($flat && !$idSubtable) {
$expanded = true;
}
$dataTable = self::getDataTableFromArchive($recordName, $idSite, $period, $date, $segment, $expanded, $idSubtable, $depth);
$dataTable->queueFilter('ReplaceColumnNames');
if ($expanded) {
$dataTable->queueFilterSubtables('ReplaceColumnNames');
}
if ($flat) {
$dataTable->disableRecursiveFilters();
}
return $dataTable;
}
private function getSiteIdsThatAreRequestedInThisArchiveButWereNotInvalidatedYet()
{
if (is_null(self::$cache)) {
self::$cache = Cache::getTransientCache();
}
$id = 'Archive.SiteIdsOfRememberedReportsInvalidated';
if (!self::$cache->contains($id)) {
self::$cache->save($id, array());
}
$siteIdsAlreadyHandled = self::$cache->fetch($id);
$siteIdsRequested = $this->params->getIdSites();
foreach ($siteIdsRequested as $index => $siteIdRequested) {
$siteIdRequested = (int) $siteIdRequested;
if (in_array($siteIdRequested, $siteIdsAlreadyHandled)) {
unset($siteIdsRequested[$index]); // was already handled previously, do not do it again
} else {
$siteIdsAlreadyHandled[] = $siteIdRequested; // we will handle this id this time
}
}
self::$cache->save($id, $siteIdsAlreadyHandled);
return $siteIdsRequested;
}
private function invalidatedReportsIfNeeded()
{
$siteIdsRequested = $this->getSiteIdsThatAreRequestedInThisArchiveButWereNotInvalidatedYet();
if (empty($siteIdsRequested)) {
return; // all requested site ids were already handled
}
$sitesPerDays = $this->invalidator->getRememberedArchivedReportsThatShouldBeInvalidated();
foreach ($sitesPerDays as $date => $siteIds) {
if (empty($siteIds)) {
continue;
}
$siteIdsToActuallyInvalidate = array_intersect($siteIds, $siteIdsRequested);
if (empty($siteIdsToActuallyInvalidate)) {
continue; // all site ids that should be handled are already handled
}
try {
$this->invalidator->markArchivesAsInvalidated($siteIdsToActuallyInvalidate, array(Date::factory($date)), false);
} catch (\Exception $e) {
Site::clearCache();
throw $e;
}
}
Site::clearCache();
}
/**
@ -477,7 +578,7 @@ class Archive
* @param null|int $idSubtable
* @return Archive\DataCollection
*/
private function get($archiveNames, $archiveDataType, $idSubtable = null)
protected function get($archiveNames, $archiveDataType, $idSubtable = null)
{
if (!is_array($archiveNames)) {
$archiveNames = array($archiveNames);
@ -487,35 +588,41 @@ class Archive
if ($idSubtable !== null
&& $idSubtable != self::ID_SUBTABLE_LOAD_ALL_SUBTABLES
) {
foreach ($archiveNames as &$name) {
$name = $this->appendIdsubtable($name, $idSubtable);
// this is also done in ArchiveSelector. It should be actually only done in ArchiveSelector but DataCollection
// does require to have the subtableId appended. Needs to be changed in refactoring to have it only in one
// place.
$dataNames = array();
foreach ($archiveNames as $name) {
$dataNames[] = ArchiveSelector::appendIdsubtable($name, $idSubtable);
}
} else {
$dataNames = $archiveNames;
}
$result = new Archive\DataCollection(
$archiveNames, $archiveDataType, $this->params->getIdSites(), $this->params->getPeriods(), $defaultRow = null);
$dataNames, $archiveDataType, $this->params->getIdSites(), $this->params->getPeriods(), $defaultRow = null);
$archiveIds = $this->getArchiveIds($archiveNames);
if (empty($archiveIds)) {
return $result;
}
$loadAllSubtables = $idSubtable == self::ID_SUBTABLE_LOAD_ALL_SUBTABLES;
$archiveData = ArchiveSelector::getArchiveData($archiveIds, $archiveNames, $archiveDataType, $loadAllSubtables);
$archiveData = ArchiveSelector::getArchiveData($archiveIds, $archiveNames, $archiveDataType, $idSubtable);
$isNumeric = $archiveDataType == 'numeric';
foreach ($archiveData as $row) {
// values are grouped by idsite (site ID), date1-date2 (date range), then name (field name)
$idSite = $row['idsite'];
$periodStr = $row['date1'] . "," . $row['date2'];
$periodStr = $row['date1'] . ',' . $row['date2'];
if ($archiveDataType == 'numeric') {
$value = $this->formatNumericValue($row['value']);
if ($isNumeric) {
$row['value'] = $this->formatNumericValue($row['value']);
} else {
$value = $this->uncompress($row['value']);
$result->addMetadata($idSite, $periodStr, 'ts_archived', $row['ts_archived']);
$result->addMetadata($row['idsite'], $periodStr, DataTable::ARCHIVED_DATE_METADATA_NAME, $row['ts_archived']);
}
$resultRow = & $result->get($idSite, $periodStr);
$resultRow[$row['name']] = $value;
$result->set($row['idsite'], $periodStr, $row['name'], $row['value']);
}
return $result;
@ -526,6 +633,9 @@ class Archive
* queried. This function will use the idarchive cache if it has the right data,
* query archive tables for IDs w/o launching archiving, or launch archiving and
* get the idarchive from ArchiveProcessor instances.
*
* @param string $archiveNames
* @return array
*/
private function getArchiveIds($archiveNames)
{
@ -533,7 +643,7 @@ class Archive
// figure out which archives haven't been processed (if an archive has been processed,
// then we have the archive IDs in $this->idarchives)
$doneFlags = array();
$doneFlags = array();
$archiveGroups = array();
foreach ($plugins as $plugin) {
$doneFlag = $this->getDoneStringForPlugin($plugin);
@ -542,7 +652,7 @@ class Archive
if (!isset($this->idarchives[$doneFlag])) {
$archiveGroup = $this->getArchiveGroupOfPlugin($plugin);
if($archiveGroup == self::ARCHIVE_ALL_PLUGINS_FLAG) {
if ($archiveGroup == self::ARCHIVE_ALL_PLUGINS_FLAG) {
$archiveGroup = reset($plugins);
}
$archiveGroups[] = $archiveGroup;
@ -560,19 +670,7 @@ class Archive
}
}
// order idarchives by the table month they belong to
$idArchivesByMonth = array();
foreach (array_keys($doneFlags) as $doneFlag) {
if (empty($this->idarchives[$doneFlag])) {
continue;
}
foreach ($this->idarchives[$doneFlag] as $dateRange => $idarchives) {
foreach ($idarchives as $id) {
$idArchivesByMonth[$dateRange][] = $id;
}
}
}
$idArchivesByMonth = $this->getIdArchivesByMonth($doneFlags);
return $idArchivesByMonth;
}
@ -587,6 +685,8 @@ class Archive
*/
private function cacheArchiveIdsAfterLaunching($archiveGroups, $plugins)
{
$this->invalidatedReportsIfNeeded();
$today = Date::today();
foreach ($this->params->getPeriods() as $period) {
@ -600,14 +700,14 @@ class Archive
// we already know there are no stats for this period
// we add one day to make sure we don't miss the day of the website creation
if ($twoDaysAfterPeriod->isEarlier($site->getCreationDate())) {
Log::verbose("Archive site %s, %s (%s) skipped, archive is before the website was created.",
Log::debug("Archive site %s, %s (%s) skipped, archive is before the website was created.",
$idSite, $period->getLabel(), $period->getPrettyString());
continue;
}
// if the starting date is in the future we know there is no visiidsite = ?t
if ($twoDaysBeforePeriod->isLater($today)) {
Log::verbose("Archive site %s, %s (%s) skipped, archive is after today.",
Log::debug("Archive site %s, %s (%s) skipped, archive is after today.",
$idSite, $period->getLabel(), $period->getPrettyString());
continue;
}
@ -627,7 +727,7 @@ class Archive
private function cacheArchiveIdsWithoutLaunching($plugins)
{
$idarchivesByReport = ArchiveSelector::getArchiveIds(
$this->params->getIdSites(), $this->params->getPeriods(), $this->params->getSegment(), $plugins, $this->params->isSkipAggregationOfSubTables());
$this->params->getIdSites(), $this->params->getPeriods(), $this->params->getSegment(), $plugins);
// initialize archive ID cache for each report
foreach ($plugins as $plugin) {
@ -655,8 +755,7 @@ class Archive
$this->params->getIdSites(),
$this->params->getSegment(),
$this->getPeriodLabel(),
$plugin,
$this->params->isSkipAggregationOfSubTables()
$plugin
);
}
@ -707,11 +806,6 @@ class Archive
return round((float)$value, 2);
}
private function uncompress($data)
{
return @gzuncompress($data);
}
/**
* Initializes the archive ID cache ($this->idarchives) for a particular 'done' flag.
*
@ -723,6 +817,8 @@ class Archive
* If this function is not called, then periods with no visits will not add
* entries to the cache. If the archive is used again, SQL will be executed to
* try and find the archive IDs even though we know there are none.
*
* @param string $doneFlag
*/
private function initializeArchiveIdCache($doneFlag)
{
@ -745,7 +841,10 @@ class Archive
*/
private function getArchiveGroupOfPlugin($plugin)
{
if ($this->getPeriodLabel() != 'range') {
$periods = $this->params->getPeriods();
$periodLabel = reset($periods)->getLabel();
if (Rules::shouldProcessReportsAllPlugins($this->params->getIdSites(), $this->params->getSegment(), $periodLabel)) {
return self::ARCHIVE_ALL_PLUGINS_FLAG;
}
@ -755,7 +854,7 @@ class Archive
/**
* Returns the name of the plugin that archives a given report.
*
* @param string $report Archive data name, eg, `'nb_visits'`, `'UserSettings_...'`, etc.
* @param string $report Archive data name, eg, `'nb_visits'`, `'DevicesDetection_...'`, etc.
* @return string Plugin name.
* @throws \Exception If a plugin cannot be found or if the plugin for the report isn't
* activated.
@ -766,9 +865,9 @@ class Archive
if (in_array($report, Metrics::getVisitsMetricNames())) {
$report = 'VisitsSummary_CoreMetrics';
} // Goal_* metrics are processed by the Goals plugin (HACK)
else if (strpos($report, 'Goal_') === 0) {
elseif (strpos($report, 'Goal_') === 0) {
$report = 'Goals_Metrics';
} else if (strrpos($report, '_returning') === strlen($report) - strlen('_returning')) { // HACK
} elseif (strrpos($report, '_returning') === strlen($report) - strlen('_returning')) { // HACK
$report = 'VisitFrequency_Metrics';
}
@ -789,7 +888,7 @@ class Archive
*/
private function prepareArchive(array $archiveGroups, Site $site, Period $period)
{
$parameters = new ArchiveProcessor\Parameters($site, $period, $this->params->getSegment(), $this->params->isSkipAggregationOfSubTables());
$parameters = new ArchiveProcessor\Parameters($site, $period, $this->params->getSegment());
$archiveLoader = new ArchiveProcessor\Loader($parameters);
$periodString = $period->getRangeString();
@ -801,9 +900,37 @@ class Archive
$idArchive = $archiveLoader->prepareArchive($plugin);
if($idArchive) {
if ($idArchive) {
$this->idarchives[$doneFlag][$periodString][] = $idArchive;
}
}
}
private function getIdArchivesByMonth($doneFlags)
{
// order idarchives by the table month they belong to
$idArchivesByMonth = array();
foreach (array_keys($doneFlags) as $doneFlag) {
if (empty($this->idarchives[$doneFlag])) {
continue;
}
foreach ($this->idarchives[$doneFlag] as $dateRange => $idarchives) {
foreach ($idarchives as $id) {
$idArchivesByMonth[$dateRange][] = $id;
}
}
}
return $idArchivesByMonth;
}
/**
* @internal
*/
public static function clearStaticCache()
{
self::$cache = null;
}
}

317
www/analytics/core/Archive/ArchiveInvalidator.php Normal file
View file

@ -0,0 +1,317 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\Archive\ArchiveInvalidator\InvalidationResult;
use Piwik\CronArchive\SitesToReprocessDistributedList;
use Piwik\DataAccess\ArchiveTableCreator;
use Piwik\DataAccess\Model;
use Piwik\Date;
use Piwik\Option;
use Piwik\Plugins\CoreAdminHome\Tasks\ArchivesToPurgeDistributedList;
use Piwik\Plugins\PrivacyManager\PrivacyManager;
use Piwik\Period;
use Piwik\Segment;
/**
* Service that can be used to invalidate archives or add archive references to a list so they will
* be invalidated later.
*
* Archives are put in an "invalidated" state by setting the done flag to `ArchiveWriter::DONE_INVALIDATED`.
* This class also adds the archive's associated site to the a distributed list and adding the archive's year month to another
* distributed list.
*
* CronArchive will reprocess the archive data for all sites in the first list, and a scheduled task
* will purge the old, invalidated data in archive tables identified by the second list.
*
* Until CronArchive, or browser triggered archiving, re-processes data for an invalidated archive, the invalidated
* archive data will still be displayed in the UI and API.
*
* ### Deferred Invalidation
*
* Invalidating archives means running queries on one or more archive tables. In some situations, like during
* tracking, this is not desired. In such cases, archive references can be added to a list via the
* rememberToInvalidateArchivedReportsLater method, which will add the reference to a distributed list
*
* Later, during Piwik's normal execution, the list will be read and every archive it references will
* be invalidated.
*/
class ArchiveInvalidator
{
private $rememberArchivedReportIdStart = 'report_to_invalidate_';
/**
* @var Model
*/
private $model;
public function __construct(Model $model)
{
$this->model = $model;
}
public function rememberToInvalidateArchivedReportsLater($idSite, Date $date)
{
$key = $this->buildRememberArchivedReportId($idSite, $date->toString());
$value = Option::get($key);
// we do not really have to get the value first. we could simply always try to call set() and it would update or
// insert the record if needed but we do not want to lock the table (especially since there are still some
// MyISAM installations)
if (false === $value) {
Option::set($key, '1');
}
}
public function getRememberedArchivedReportsThatShouldBeInvalidated()
{
$reports = Option::getLike($this->rememberArchivedReportIdStart . '%_%');
$sitesPerDay = array();
foreach ($reports as $report => $value) {
$report = str_replace($this->rememberArchivedReportIdStart, '', $report);
$report = explode('_', $report);
$siteId = (int) $report[0];
$date = $report[1];
if (empty($sitesPerDay[$date])) {
$sitesPerDay[$date] = array();
}
$sitesPerDay[$date][] = $siteId;
}
return $sitesPerDay;
}
private function buildRememberArchivedReportId($idSite, $date)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite);
$id .= '_' . trim($date);
return $id;
}
private function buildRememberArchivedReportIdForSite($idSite)
{
return $this->rememberArchivedReportIdStart . (int) $idSite;
}
public function forgetRememberedArchivedReportsToInvalidateForSite($idSite)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite) . '_%';
Option::deleteLike($id);
}
/**
* @internal
*/
public function forgetRememberedArchivedReportsToInvalidate($idSite, Date $date)
{
$id = $this->buildRememberArchivedReportId($idSite, $date->toString());
Option::delete($id);
}
/**
* @param $idSites int[]
* @param $dates Date[]
* @param $period string
* @param $segment Segment
* @param bool $cascadeDown
* @return InvalidationResult
* @throws \Exception
*/
public function markArchivesAsInvalidated(array $idSites, array $dates, $period, Segment $segment = null, $cascadeDown = false)
{
$invalidationInfo = new InvalidationResult();
$datesToInvalidate = $this->removeDatesThatHaveBeenPurged($dates, $invalidationInfo);
if (empty($period)) {
// if the period is empty, we don't need to cascade in any way, since we'll remove all periods
$periodDates = $this->getDatesByYearMonthAndPeriodType($dates);
} else {
$periods = $this->getPeriodsToInvalidate($datesToInvalidate, $period, $cascadeDown);
$periodDates = $this->getPeriodDatesByYearMonthAndPeriodType($periods);
}
$periodDates = $this->getUniqueDates($periodDates);
$this->markArchivesInvalidated($idSites, $periodDates, $segment);
$yearMonths = array_keys($periodDates);
$this->markInvalidatedArchivesForReprocessAndPurge($idSites, $yearMonths);
foreach ($idSites as $idSite) {
foreach ($dates as $date) {
$this->forgetRememberedArchivedReportsToInvalidate($idSite, $date);
}
}
return $invalidationInfo;
}
/**
* @param string[][][] $periodDates
* @return string[][][]
*/
private function getUniqueDates($periodDates)
{
$result = array();
foreach ($periodDates as $yearMonth => $periodsByYearMonth) {
foreach ($periodsByYearMonth as $periodType => $periods) {
$result[$yearMonth][$periodType] = array_unique($periods);
}
}
return $result;
}
/**
* @param Date[] $dates
* @param string $periodType
* @param bool $cascadeDown
* @return Period[]
*/
private function getPeriodsToInvalidate($dates, $periodType, $cascadeDown)
{
$periodsToInvalidate = array();
foreach ($dates as $date) {
if ($periodType == 'range') {
$date = $date . ',' . $date;
}
$period = Period\Factory::build($periodType, $date);
$periodsToInvalidate[] = $period;
if ($cascadeDown) {
$periodsToInvalidate = array_merge($periodsToInvalidate, $period->getAllOverlappingChildPeriods());
}
if ($periodType != 'year'
&& $periodType != 'range'
) {
$periodsToInvalidate[] = Period\Factory::build('year', $date);
}
}
return $periodsToInvalidate;
}
/**
* @param Period[] $periods
* @return string[][][]
*/
private function getPeriodDatesByYearMonthAndPeriodType($periods)
{
$result = array();
foreach ($periods as $period) {
$date = $period->getDateStart();
$periodType = $period->getId();
$yearMonth = ArchiveTableCreator::getTableMonthFromDate($date);
$result[$yearMonth][$periodType][] = $date->toString();
}
return $result;
}
/**
* Called when deleting all periods.
*
* @param Date[] $dates
* @return string[][][]
*/
private function getDatesByYearMonthAndPeriodType($dates)
{
$result = array();
foreach ($dates as $date) {
$yearMonth = ArchiveTableCreator::getTableMonthFromDate($date);
$result[$yearMonth][null][] = $date->toString();
// since we're removing all periods, we must make sure to remove year periods as well.
// this means we have to make sure the january table is processed.
$janYearMonth = $date->toString('Y') . '_01';
$result[$janYearMonth][null][] = $date->toString();
}
return $result;
}
/**
* @param int[] $idSites
* @param string[][][] $dates
* @throws \Exception
*/
private function markArchivesInvalidated($idSites, $dates, Segment $segment = null)
{
$archiveNumericTables = ArchiveTableCreator::getTablesArchivesInstalled($type = ArchiveTableCreator::NUMERIC_TABLE);
foreach ($archiveNumericTables as $table) {
$tableDate = ArchiveTableCreator::getDateFromTableName($table);
if (empty($dates[$tableDate])) {
continue;
}
$this->model->updateArchiveAsInvalidated($table, $idSites, $dates[$tableDate], $segment);
}
}
/**
* @param Date[] $dates
* @param InvalidationResult $invalidationInfo
* @return \Piwik\Date[]
*/
private function removeDatesThatHaveBeenPurged($dates, InvalidationResult $invalidationInfo)
{
$this->findOlderDateWithLogs($invalidationInfo);
$result = array();
foreach ($dates as $date) {
// we should only delete reports for dates that are more recent than N days
if ($invalidationInfo->minimumDateWithLogs
&& $date->isEarlier($invalidationInfo->minimumDateWithLogs)
) {
$invalidationInfo->warningDates[] = $date->toString();
continue;
}
$result[] = $date;
$invalidationInfo->processedDates[] = $date->toString();
}
return $result;
}
private function findOlderDateWithLogs(InvalidationResult $info)
{
// If using the feature "Delete logs older than N days"...
$purgeDataSettings = PrivacyManager::getPurgeDataSettings();
$logsDeletedWhenOlderThanDays = (int)$purgeDataSettings['delete_logs_older_than'];
$logsDeleteEnabled = $purgeDataSettings['delete_logs_enable'];
if ($logsDeleteEnabled
&& $logsDeletedWhenOlderThanDays
) {
$info->minimumDateWithLogs = Date::factory('today')->subDay($logsDeletedWhenOlderThanDays);
}
}
/**
* @param array $idSites
* @param array $yearMonths
*/
private function markInvalidatedArchivesForReprocessAndPurge(array $idSites, $yearMonths)
{
$store = new SitesToReprocessDistributedList();
$store->add($idSites);
$archivesToPurge = new ArchivesToPurgeDistributedList();
$archivesToPurge->add($yearMonths);
}
}

56
www/analytics/core/Archive/ArchiveInvalidator/InvalidationResult.php Normal file
View file

@ -0,0 +1,56 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Archive\ArchiveInvalidator;
use Piwik\Date;
/**
* Information about the result of an archive invalidation operation.
*/
class InvalidationResult
{
/**
* Dates that couldn't be invalidated because they are earlier than the configured log
* deletion limit.
*
* @var array
*/
public $warningDates = array();
/**
* Dates that were successfully invalidated.
*
* @var array
*/
public $processedDates = array();
/**
* The day of the oldest log entry.
*
* @var Date|bool
*/
public $minimumDateWithLogs = false;
/**
* @return string[]
*/
public function makeOutputLogs()
{
$output = array();
if ($this->warningDates) {
$output[] = 'Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: ' .
implode(", ", $this->warningDates) .
"\n The last day with logs is " . $this->minimumDateWithLogs . ". " .
"\n Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.";
}
$output[] = "Success. The following dates were invalidated successfully: " . implode(", ", $this->processedDates);
return $output;
}
}

272
www/analytics/core/Archive/ArchivePurger.php Normal file
View file

@ -0,0 +1,272 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Config;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveTableCreator;
use Piwik\DataAccess\Model;
use Piwik\Date;
use Piwik\Piwik;
use Psr\Log\LoggerInterface;
use Psr\Log\LogLevel;
/**
* Service that purges temporary, error-ed, invalid and custom range archives from archive tables.
*
* Temporary archives are purged if they were archived before a specific time. The time is dependent
* on whether browser triggered archiving is enabled or not.
*
* Error-ed archives are purged w/o constraint.
*
* Invalid archives are purged if a new, valid, archive exists w/ the same site, date, period combination.
* Archives are marked as invalid via Piwik\Archive\ArchiveInvalidator.
*/
class ArchivePurger
{
/**
* @var Model
*/
private $model;
/**
* Date threshold for purging custom range archives. Archives that are older than this date
* are purged unconditionally from the requested archive table.
*
* @var Date
*/
private $purgeCustomRangesOlderThan;
/**
* Date to use for 'yesterday'. Exists so tests can override this value.
*
* @var Date
*/
private $yesterday;
/**
* Date to use for 'today'. Exists so tests can override this value.
*
* @var $today
*/
private $today;
/**
* Date to use for 'now'. Exists so tests can override this value.
*
* @var int
*/
private $now;
/**
* @var LoggerInterface
*/
private $logger;
public function __construct(Model $model = null, Date $purgeCustomRangesOlderThan = null, LoggerInterface $logger = null)
{
$this->model = $model ?: new Model();
$this->purgeCustomRangesOlderThan = $purgeCustomRangesOlderThan ?: self::getDefaultCustomRangeToPurgeAgeThreshold();
$this->yesterday = Date::factory('yesterday');
$this->today = Date::factory('today');
$this->now = time();
$this->logger = $logger ?: StaticContainer::get('Psr\Log\LoggerInterface');
}
/**
* Purge all invalidate archives for whom there are newer, valid archives from the archive
* table that stores data for `$date`.
*
* @param Date $date The date identifying the archive table.
* @return int The total number of archive rows deleted (from both the blog & numeric tables).
*/
public function purgeInvalidatedArchivesFrom(Date $date)
{
$numericTable = ArchiveTableCreator::getNumericTable($date);
// we don't want to do an INNER JOIN on every row in a archive table that can potentially have tens to hundreds of thousands of rows,
// so we first look for sites w/ invalidated archives, and use this as a constraint in getInvalidatedArchiveIdsSafeToDelete() below.
// the constraint will hit an INDEX and speed up the inner join that happens in getInvalidatedArchiveIdsSafeToDelete().
$idSites = $this->model->getSitesWithInvalidatedArchive($numericTable);
if (empty($idSites)) {
$this->logger->debug("No sites with invalidated archives found in {table}.", array('table' => $numericTable));
return 0;
}
$archiveIds = $this->model->getInvalidatedArchiveIdsSafeToDelete($numericTable, $idSites);
if (empty($archiveIds)) {
$this->logger->debug("No invalidated archives found in {table} with newer, valid archives.", array('table' => $numericTable));
return 0;
}
$this->logger->info("Found {countArchiveIds} invalidated archives safe to delete in {table}.", array(
'table' => $numericTable, 'countArchiveIds' => count($archiveIds)
));
$deletedRowCount = $this->deleteArchiveIds($date, $archiveIds);
$this->logger->debug("Deleted {count} rows in {table} and its associated blob table.", array(
'table' => $numericTable, 'count' => $deletedRowCount
));
return $deletedRowCount;
}
/**
* Removes the outdated archives for the given month.
* (meaning they are marked with a done flag of ArchiveWriter::DONE_OK_TEMPORARY or ArchiveWriter::DONE_ERROR)
*
* @param Date $dateStart Only the month will be used
* @return int Returns the total number of rows deleted.
*/
public function purgeOutdatedArchives(Date $dateStart)
{
$purgeArchivesOlderThan = $this->getOldestTemporaryArchiveToKeepThreshold();
$deletedRowCount = 0;
$idArchivesToDelete = $this->getOutdatedArchiveIds($dateStart, $purgeArchivesOlderThan);
if (!empty($idArchivesToDelete)) {
$deletedRowCount = $this->deleteArchiveIds($dateStart, $idArchivesToDelete);
$this->logger->info("Deleted {count} rows in archive tables (numeric + blob) for {date}.", array(
'count' => $deletedRowCount,
'date' => $dateStart
));
} else {
$this->logger->debug("No outdated archives found in archive numeric table for {date}.", array('date' => $dateStart));
}
$this->logger->debug("Purging temporary archives: done [ purged archives older than {date} in {yearMonth} ] [Deleted IDs: {deletedIds}]", array(
'date' => $purgeArchivesOlderThan,
'yearMonth' => $dateStart->toString('Y-m'),
'deletedIds' => implode(',', $idArchivesToDelete)
));
return $deletedRowCount;
}
protected function getOutdatedArchiveIds(Date $date, $purgeArchivesOlderThan)
{
$archiveTable = ArchiveTableCreator::getNumericTable($date);
$result = $this->model->getTemporaryArchivesOlderThan($archiveTable, $purgeArchivesOlderThan);
$idArchivesToDelete = array();
if (!empty($result)) {
foreach ($result as $row) {
$idArchivesToDelete[] = $row['idarchive'];
}
}
return $idArchivesToDelete;
}
/**
* Deleting "Custom Date Range" reports after 1 day, since they can be re-processed and would take up un-necessary space.
*
* @param $date Date
* @return int The total number of rows deleted from both the numeric & blob table.
*/
public function purgeArchivesWithPeriodRange(Date $date)
{
$numericTable = ArchiveTableCreator::getNumericTable($date);
$blobTable = ArchiveTableCreator::getBlobTable($date);
$deletedCount = $this->model->deleteArchivesWithPeriod(
$numericTable, $blobTable, Piwik::$idPeriods['range'], $this->purgeCustomRangesOlderThan);
$level = $deletedCount == 0 ? LogLevel::DEBUG : LogLevel::INFO;
$this->logger->log($level, "Purged {count} range archive rows from {numericTable} & {blobTable}.", array(
'count' => $deletedCount,
'numericTable' => $numericTable,
'blobTable' => $blobTable
));
$this->logger->debug(" [ purged archives older than {threshold} ]", array('threshold' => $this->purgeCustomRangesOlderThan));
return $deletedCount;
}
/**
* Deletes by batches Archive IDs in the specified month,
*
* @param Date $date
* @param $idArchivesToDelete
* @return int Number of rows deleted from both numeric + blob table.
*/
protected function deleteArchiveIds(Date $date, $idArchivesToDelete)
{
$batches = array_chunk($idArchivesToDelete, 1000);
$numericTable = ArchiveTableCreator::getNumericTable($date);
$blobTable = ArchiveTableCreator::getBlobTable($date);
$deletedCount = 0;
foreach ($batches as $idsToDelete) {
$deletedCount += $this->model->deleteArchiveIds($numericTable, $blobTable, $idsToDelete);
}
return $deletedCount;
}
/**
* Returns a timestamp indicating outdated archives older than this timestamp (processed before) can be purged.
*
* @return int|bool Outdated archives older than this timestamp should be purged
*/
protected function getOldestTemporaryArchiveToKeepThreshold()
{
$temporaryArchivingTimeout = Rules::getTodayArchiveTimeToLive();
if (Rules::isBrowserTriggerEnabled()) {
// If Browser Archiving is enabled, it is likely there are many more temporary archives
// We delete more often which is safe, since reports are re-processed on demand
return Date::factory($this->now - 2 * $temporaryArchivingTimeout)->getDateTime();
}
// If cron core:archive command is building the reports, we should keep all temporary reports from today
return $this->yesterday->getDateTime();
}
private static function getDefaultCustomRangeToPurgeAgeThreshold()
{
$daysRangesValid = Config::getInstance()->General['purge_date_range_archives_after_X_days'];
return Date::factory('today')->subDay($daysRangesValid)->getDateTime();
}
/**
* For tests.
*
* @param Date $yesterday
*/
public function setYesterdayDate(Date $yesterday)
{
$this->yesterday = $yesterday;
}
/**
* For tests.
*
* @param Date $today
*/
public function setTodayDate(Date $today)
{
$this->today = $today;
}
/**
* For tests.
*
* @param int $now
*/
public function setNow($now)
{
$this->now = $now;
}
}

144
www/analytics/core/Archive/Chunk.php Normal file
View file

@ -0,0 +1,144 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\DataTable;
/**
* This class is used to split blobs of DataTables into chunks. Each blob used to be stored under one blob in the
* archive table. For better efficiency we do now combine multiple DataTable into one blob entry.
*
* Chunks are identified by having the recordName $recordName_chunk_0_99, $recordName_chunk_100_199 (this chunk stores
* the subtable 100-199).
*/
class Chunk
{
const ARCHIVE_APPENDIX_SUBTABLES = 'chunk';
const NUM_TABLES_IN_CHUNK = 100;
/**
* Get's the record name to use for a given tableId/subtableId.
*
* @param string $recordName eg 'Actions_ActionsUrl'
* @param int $tableId eg '5' for tableId '5'
* @return string eg 'Actions_ActionsUrl_chunk_0_99' as the table should be stored under this blob id.
*/
public function getRecordNameForTableId($recordName, $tableId)
{
$chunk = (floor($tableId / self::NUM_TABLES_IN_CHUNK));
$start = $chunk * self::NUM_TABLES_IN_CHUNK;
$end = $start + self::NUM_TABLES_IN_CHUNK - 1;
return $recordName . $this->getAppendix() . $start . '_' . $end;
}
/**
* Moves the given blobs into chunks and assigns a proper record name containing the chunk number.
*
* @param string $recordName The original archive record name, eg 'Actions_ActionsUrl'
* @param array $blobs An array containg a mapping of tableIds to blobs. Eg array(0 => 'blob', 1 => 'subtableBlob', ...)
* @return array An array where each blob is moved into a chunk, indexed by recordNames.
* eg array('Actions_ActionsUrl_chunk_0_99' => array(0 => 'blob', 1 => 'subtableBlob', ...),
* 'Actions_ActionsUrl_chunk_100_199' => array(...))
*/
public function moveArchiveBlobsIntoChunks($recordName, $blobs)
{
$chunks = array();
foreach ($blobs as $tableId => $blob) {
$name = $this->getRecordNameForTableId($recordName, $tableId);
if (!array_key_exists($name, $chunks)) {
$chunks[$name] = array();
}
$chunks[$name][$tableId] = $blob;
}
return $chunks;
}
/**
* Detects whether a recordName like 'Actions_ActionUrls_chunk_0_99' or 'Actions_ActionUrls' belongs to a
* chunk or not.
*
* To be a valid recordName that belongs to a chunk it must end with '_chunk_NUMERIC_NUMERIC'.
*
* @param string $recordName
* @return bool
*/
public function isRecordNameAChunk($recordName)
{
$posAppendix = $this->getEndPosOfChunkAppendix($recordName);
if (false === $posAppendix) {
return false;
}
// will contain "0_99" of "chunk_0_99"
$blobId = substr($recordName, $posAppendix);
return $this->isChunkRange($blobId);
}
private function isChunkRange($blobId)
{
$blobId = explode('_', $blobId);
return 2 === count($blobId) && is_numeric($blobId[0]) && is_numeric($blobId[1]);
}
/**
* When having a record like 'Actions_ActionUrls_chunk_0_99" it will return the raw recordName 'Actions_ActionUrls'.
*
* @param string $recordName
* @return string
*/
public function getRecordNameWithoutChunkAppendix($recordName)
{
if (!$this->isRecordNameAChunk($recordName)) {
return $recordName;
}
$posAppendix = $this->getStartPosOfChunkAppendix($recordName);
if (false === $posAppendix) {
return $recordName;
}
return substr($recordName, 0, $posAppendix);
}
/**
* Returns the string that is appended to the original record name. This appendix identifes a record name is a
* chunk.
* @return string
*/
public function getAppendix()
{
return '_' . self::ARCHIVE_APPENDIX_SUBTABLES . '_';
}
private function getStartPosOfChunkAppendix($recordName)
{
return strpos($recordName, $this->getAppendix());
}
private function getEndPosOfChunkAppendix($recordName)
{
$pos = strpos($recordName, $this->getAppendix());
if ($pos === false) {
return false;
}
return $pos + strlen($this->getAppendix());
}
}

49
www/analytics/core/Archive/DataCollection.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -137,6 +137,21 @@ class DataCollection
return $this->data[$idSite][$period];
}
/**
* Set data for a specific site & period. If there is no data for the given site ID & period,
* it is set to the default row.
*
* @param int $idSite
* @param string $period eg, '2012-01-01,2012-01-31'
* @param string $name eg 'nb_visits'
* @param string $value eg 5
*/
public function set($idSite, $period, $name, $value)
{
$row = & $this->get($idSite, $period);
$row[$name] = $value;
}
/**
* Adds a new metadata to the data for specific site & period. If there is no
* data for the given site ID & period, it is set to the default row.
@ -188,6 +203,7 @@ class DataCollection
$this->putRowInIndex($result, $indexKeys, $row, $idSite, $period);
}
}
return $result;
}
@ -208,9 +224,27 @@ class DataCollection
$this->dataNames, $this->dataType, $this->sitesId, $this->periods, $this->defaultRow);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->make($index, $resultIndices);
}
/**
* See {@link DataTableFactory::makeMerged()}
*
* @param array $resultIndices
* @return DataTable|DataTable\Map
* @throws Exception
*/
public function getMergedDataTable($resultIndices)
{
$dataTableFactory = new DataTableFactory(
$this->dataNames, $this->dataType, $this->sitesId, $this->periods, $this->defaultRow);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->makeMerged($index, $resultIndices);
}
/**
* Returns archive data as a DataTable indexed by metadata. Indexed data will
* be represented by Map instances. Each DataTable will have
@ -249,6 +283,7 @@ class DataCollection
$dataTableFactory->useSubtable($idSubTable);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->make($index, $resultIndices);
}
@ -296,12 +331,16 @@ class DataCollection
if ($metadataName == DataTableFactory::TABLE_METADATA_SITE_INDEX) {
$indexKeyValues = array_values($this->sitesId);
} else if ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
} elseif ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
$indexKeyValues = array_keys($this->periods);
}
foreach ($indexKeyValues as $key) {
$result[$key] = $this->createOrderedIndex($metadataNamesToIndexBy);
if (empty($metadataNamesToIndexBy)) {
$result = array_fill_keys($indexKeyValues, array());
} else {
foreach ($indexKeyValues as $key) {
$result[$key] = $this->createOrderedIndex($metadataNamesToIndexBy);
}
}
}
@ -318,7 +357,7 @@ class DataCollection
foreach ($metadataNamesToIndexBy as $metadataName) {
if ($metadataName == DataTableFactory::TABLE_METADATA_SITE_INDEX) {
$key = $idSite;
} else if ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
} elseif ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
$key = $period;
} else {
$key = $row[self::METADATA_CONTAINER_ROW_KEY][$metadataName];

243
www/analytics/core/Archive/DataTableFactory.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -95,6 +95,23 @@ class DataTableFactory
$this->defaultRow = $defaultRow;
}
/**
* Returns the ID of the site a table is related to based on the 'site' metadata entry,
* or null if there is none.
*
* @param DataTable $table
* @return int|null
*/
public static function getSiteIdFromMetadata(DataTable $table)
{
$site = $table->getMetadata('site');
if (empty($site)) {
return null;
} else {
return $site->getId();
}
}
/**
* Tells the factory instance to expand the DataTables that are created by
* creating subtables and setting the subtable IDs of rows w/ subtables correctly.
@ -128,6 +145,11 @@ class DataTableFactory
$this->idSubtable = $idSubtable;
}
private function isNumericDataType()
{
return $this->dataType == 'numeric';
}
/**
* Creates a DataTable|Set instance using an index of
* archive data.
@ -139,21 +161,63 @@ class DataTableFactory
*/
public function make($index, $resultIndices)
{
$keyMetadata = $this->getDefaultMetadata();
if (empty($resultIndices)) {
// for numeric data, if there's no index (and thus only 1 site & period in the query),
// we want to display every queried metric name
if (empty($index)
&& $this->dataType == 'numeric'
&& $this->isNumericDataType()
) {
$index = $this->defaultRow;
}
$dataTable = $this->createDataTable($index, $keyMetadata = array());
$dataTable = $this->createDataTable($index, $keyMetadata);
} else {
$dataTable = $this->createDataTableMapFromIndex($index, $resultIndices, $keyMetadata = array());
$dataTable = $this->createDataTableMapFromIndex($index, $resultIndices, $keyMetadata);
}
return $dataTable;
}
/**
* Creates a merged DataTable|Map instance using an index of archive data similar to {@link make()}.
*
* Whereas {@link make()} creates a Map for each result index (period and|or site), this will only create a Map
* for a period result index and move all site related indices into one dataTable. This is the same as doing
* `$dataTableFactory->make()->mergeChildren()` just much faster. It is mainly useful for reports across many sites
* eg `MultiSites.getAll`. Was done as part of https://github.com/piwik/piwik/issues/6809
*
* @param array $index @see DataCollection
* @param array $resultIndices an array mapping metadata names with pretty metadata labels.
*
* @return DataTable|DataTable\Map
* @throws \Exception
*/
public function makeMerged($index, $resultIndices)
{
if (!$this->isNumericDataType()) {
throw new \Exception('This method is supposed to work with non-numeric data types but it is not tested. To use it, remove this exception and write tests to be sure it works.');
}
$hasSiteIndex = isset($resultIndices[self::TABLE_METADATA_SITE_INDEX]);
$hasPeriodIndex = isset($resultIndices[self::TABLE_METADATA_PERIOD_INDEX]);
$isNumeric = $this->isNumericDataType();
// to be backwards compatible use a Simple table if needed as it will be formatted differently
$useSimpleDataTable = !$hasSiteIndex && $isNumeric;
if (!$hasSiteIndex) {
$firstIdSite = reset($this->sitesId);
$index = array($firstIdSite => $index);
}
if ($hasPeriodIndex) {
$dataTable = $this->makeMergedTableWithPeriodAndSiteIndex($index, $resultIndices, $useSimpleDataTable, $isNumeric);
} else {
$dataTable = $this->makeMergedWithSiteIndex($index, $useSimpleDataTable, $isNumeric);
}
$this->transformMetadata($dataTable);
return $dataTable;
}
@ -171,16 +235,16 @@ class DataTableFactory
* @param array $blobRow
* @return DataTable|DataTable\Map
*/
private function makeFromBlobRow($blobRow)
private function makeFromBlobRow($blobRow, $keyMetadata)
{
if ($blobRow === false) {
return new DataTable();
}
if (count($this->dataNames) === 1) {
return $this->makeDataTableFromSingleBlob($blobRow);
return $this->makeDataTableFromSingleBlob($blobRow, $keyMetadata);
} else {
return $this->makeIndexedByRecordNameDataTable($blobRow);
return $this->makeIndexedByRecordNameDataTable($blobRow, $keyMetadata);
}
}
@ -192,7 +256,7 @@ class DataTableFactory
* @param array $blobRow
* @return DataTable
*/
private function makeDataTableFromSingleBlob($blobRow)
private function makeDataTableFromSingleBlob($blobRow, $keyMetadata)
{
$recordName = reset($this->dataNames);
if ($this->idSubtable !== null) {
@ -206,7 +270,7 @@ class DataTableFactory
}
// set table metadata
$table->setMetadataValues(DataCollection::getDataRowMetadata($blobRow));
$table->setAllTableMetadata(array_merge(DataCollection::getDataRowMetadata($blobRow), $keyMetadata));
if ($this->expandDataTable) {
$table->enableRecursiveFilters();
@ -223,12 +287,12 @@ class DataTableFactory
* @param array $blobRow
* @return DataTable\Map
*/
private function makeIndexedByRecordNameDataTable($blobRow)
private function makeIndexedByRecordNameDataTable($blobRow, $keyMetadata)
{
$table = new DataTable\Map();
$table->setKeyName('recordName');
$tableMetadata = DataCollection::getDataRowMetadata($blobRow);
$tableMetadata = array_merge(DataCollection::getDataRowMetadata($blobRow), $keyMetadata);
foreach ($blobRow as $name => $blob) {
$newTable = DataTable::fromSerializedArray($blob);
@ -248,23 +312,23 @@ class DataTableFactory
* @param array $keyMetadata The metadata to add to the table when it's created.
* @return DataTable\Map
*/
private function createDataTableMapFromIndex($index, $resultIndices, $keyMetadata = array())
private function createDataTableMapFromIndex($index, $resultIndices, $keyMetadata)
{
$resultIndexLabel = reset($resultIndices);
$result = new DataTable\Map();
$result->setKeyName(reset($resultIndices));
$resultIndex = key($resultIndices);
array_shift($resultIndices);
$result = new DataTable\Map();
$result->setKeyName($resultIndexLabel);
$hasIndices = !empty($resultIndices);
foreach ($index as $label => $value) {
$keyMetadata[$resultIndex] = $label;
$keyMetadata[$resultIndex] = $this->createTableIndexMetadata($resultIndex, $label);
if (empty($resultIndices)) {
$newTable = $this->createDataTable($value, $keyMetadata);
} else {
if ($hasIndices) {
$newTable = $this->createDataTableMapFromIndex($value, $resultIndices, $keyMetadata);
} else {
$newTable = $this->createDataTable($value, $keyMetadata);
}
$result->addTable($newTable, $this->prettifyIndexLabel($resultIndex, $label));
@ -273,6 +337,15 @@ class DataTableFactory
return $result;
}
private function createTableIndexMetadata($resultIndex, $label)
{
if ($resultIndex === DataTableFactory::TABLE_METADATA_SITE_INDEX) {
return new Site($label);
} elseif ($resultIndex === DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
return $this->periods[$label];
}
}
/**
* Creates a DataTable instance from an index row.
*
@ -283,11 +356,11 @@ class DataTableFactory
private function createDataTable($data, $keyMetadata)
{
if ($this->dataType == 'blob') {
$result = $this->makeFromBlobRow($data);
$result = $this->makeFromBlobRow($data, $keyMetadata);
} else {
$result = $this->makeFromMetricsArray($data);
$result = $this->makeFromMetricsArray($data, $keyMetadata);
}
$this->setTableMetadata($keyMetadata, $result);
return $result;
}
@ -307,7 +380,7 @@ class DataTableFactory
&& $treeLevel >= $this->maxSubtableDepth
) {
// unset the subtables so DataTableManager doesn't throw
foreach ($dataTable->getRows() as $row) {
foreach ($dataTable->getRowsWithoutSummaryRow() as $row) {
$row->removeSubtable();
}
@ -316,7 +389,7 @@ class DataTableFactory
$dataName = reset($this->dataNames);
foreach ($dataTable->getRows() as $row) {
foreach ($dataTable->getRowsWithoutSummaryRow() as $row) {
$sid = $row->getIdSubDataTable();
if ($sid === null) {
continue;
@ -340,17 +413,12 @@ class DataTableFactory
}
}
/**
* Converts site IDs and period string ranges into Site instances and
* Period instances in DataTable metadata.
*/
private function transformMetadata($table)
private function getDefaultMetadata()
{
$periods = $this->periods;
$table->filter(function ($table) use ($periods) {
$table->setMetadata(DataTableFactory::TABLE_METADATA_SITE_INDEX, new Site($table->getMetadata(DataTableFactory::TABLE_METADATA_SITE_INDEX)));
$table->setMetadata(DataTableFactory::TABLE_METADATA_PERIOD_INDEX, $periods[$table->getMetadata(DataTableFactory::TABLE_METADATA_PERIOD_INDEX)]);
});
return array(
DataTableFactory::TABLE_METADATA_SITE_INDEX => new Site(reset($this->sitesId)),
DataTableFactory::TABLE_METADATA_PERIOD_INDEX => reset($this->periods),
);
}
/**
@ -368,39 +436,16 @@ class DataTableFactory
return $label;
}
/**
* @param $keyMetadata
* @param $result
*/
private function setTableMetadata($keyMetadata, $result)
{
if (!isset($keyMetadata[DataTableFactory::TABLE_METADATA_SITE_INDEX])) {
$keyMetadata[DataTableFactory::TABLE_METADATA_SITE_INDEX] = reset($this->sitesId);
}
if (!isset($keyMetadata[DataTableFactory::TABLE_METADATA_PERIOD_INDEX])) {
reset($this->periods);
$keyMetadata[DataTableFactory::TABLE_METADATA_PERIOD_INDEX] = key($this->periods);
}
// Note: $result can be a DataTable\Map
$result->filter(function ($table) use ($keyMetadata) {
foreach ($keyMetadata as $name => $value) {
$table->setMetadata($name, $value);
}
});
}
/**
* @param $data
* @return DataTable\Simple
*/
private function makeFromMetricsArray($data)
private function makeFromMetricsArray($data, $keyMetadata)
{
$table = new DataTable\Simple();
if (!empty($data)) {
$table->setAllTableMetadata(DataCollection::getDataRowMetadata($data));
$table->setAllTableMetadata(array_merge(DataCollection::getDataRowMetadata($data), $keyMetadata));
DataCollection::removeMetadataFromDataRow($data);
@ -412,15 +457,89 @@ class DataTableFactory
// w/o this code, an empty array would be created, and other parts of Piwik
// would break.
if (count($this->dataNames) == 1
&& $this->dataType == 'numeric'
&& $this->isNumericDataType()
) {
$name = reset($this->dataNames);
$table->addRow(new Row(array(Row::COLUMNS => array($name => 0))));
}
$table->setAllTableMetadata($keyMetadata);
}
$result = $table;
return $result;
}
}
private function makeMergedTableWithPeriodAndSiteIndex($index, $resultIndices, $useSimpleDataTable, $isNumeric)
{
$map = new DataTable\Map();
$map->setKeyName($resultIndices[self::TABLE_METADATA_PERIOD_INDEX]);
// we save all tables of the map in this array to be able to add rows fast
$tables = array();
foreach ($this->periods as $range => $period) {
// as the resulting table is "merged", we do only set Period metedata and no metadata for site. Instead each
// row will have an idsite metadata entry.
$metadata = array(self::TABLE_METADATA_PERIOD_INDEX => $period);
if ($useSimpleDataTable) {
$table = new DataTable\Simple();
} else {
$table = new DataTable();
}
$table->setAllTableMetadata($metadata);
$map->addTable($table, $this->prettifyIndexLabel(self::TABLE_METADATA_PERIOD_INDEX, $range));
$tables[$range] = $table;
}
foreach ($index as $idsite => $table) {
$rowMeta = array('idsite' => $idsite);
foreach ($table as $range => $row) {
if (!empty($row)) {
$tables[$range]->addRow(new Row(array(
Row::COLUMNS => $row,
Row::METADATA => $rowMeta)
));
} elseif ($isNumeric) {
$tables[$range]->addRow(new Row(array(
Row::COLUMNS => $this->defaultRow,
Row::METADATA => $rowMeta)
));
}
}
}
return $map;
}
private function makeMergedWithSiteIndex($index, $useSimpleDataTable, $isNumeric)
{
if ($useSimpleDataTable) {
$table = new DataTable\Simple();
} else {
$table = new DataTable();
}
$table->setAllTableMetadata(array(DataTableFactory::TABLE_METADATA_PERIOD_INDEX => reset($this->periods)));
foreach ($index as $idsite => $row) {
if (!empty($row)) {
$table->addRow(new Row(array(
Row::COLUMNS => $row,
Row::METADATA => array('idsite' => $idsite))
));
} elseif ($isNumeric) {
$table->addRow(new Row(array(
Row::COLUMNS => $this->defaultRow,
Row::METADATA => array('idsite' => $idsite))
));
}
}
return $table;
}
}

18
www/analytics/core/Archive/Parameters.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,7 +9,6 @@
namespace Piwik\Archive;
use Exception;
use Piwik\Period;
use Piwik\Segment;
@ -36,22 +35,16 @@ class Parameters
*/
private $segment;
/**
* @var bool
*/
private $skipAggregationOfSubTables;
public function getSegment()
{
return $this->segment;
}
public function __construct($idSites, $periods, Segment $segment, $skipAggregationOfSubTables)
public function __construct($idSites, $periods, Segment $segment)
{
$this->idSites = $idSites;
$this->periods = $periods;
$this->segment = $segment;
$this->skipAggregationOfSubTables = $skipAggregationOfSubTables;
}
public function getPeriods()
@ -63,11 +56,4 @@ class Parameters
{
return $this->idSites;
}
public function isSkipAggregationOfSubTables()
{
return $this->skipAggregationOfSubTables;
}
}

260
www/analytics/core/ArchiveProcessor.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,82 +9,80 @@
namespace Piwik;
use Exception;
use Piwik\Archive\DataTableFactory;
use Piwik\ArchiveProcessor\Parameters;
use Piwik\ArchiveProcessor\Rules;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\DataAccess\LogAggregator;
use Piwik\DataTable\Manager;
use Piwik\DataTable\Map;
use Piwik\DataTable\Row;
use Piwik\Db;
use Piwik\Period;
/**
* Used by {@link Piwik\Plugin\Archiver} instances to insert and aggregate archive data.
*
*
* ### See also
*
*
* - **{@link Piwik\Plugin\Archiver}** - to learn how plugins should implement their own analytics
* aggregation logic.
* - **{@link Piwik\DataAccess\LogAggregator}** - to learn how plugins can perform data aggregation
* across Piwik's log tables.
*
*
* ### Examples
*
*
* **Inserting numeric data**
*
*
* // function in an Archiver descendant
* public function aggregateDayReport()
* {
* $archiveProcessor = $this->getProcessor();
*
*
* $myFancyMetric = // ... calculate the metric value ...
* $archiveProcessor->insertNumericRecord('MyPlugin_myFancyMetric', $myFancyMetric);
* }
*
*
* **Inserting serialized DataTables**
*
*
* // function in an Archiver descendant
* public function aggregateDayReport()
* {
* $archiveProcessor = $this->getProcessor();
*
*
* $maxRowsInTable = Config::getInstance()->General['datatable_archiving_maximum_rows_standard'];j
*
*
* $dataTable = // ... build by aggregating visits ...
* $serializedData = $dataTable->getSerialized($maxRowsInTable, $maxRowsInSubtable = $maxRowsInTable,
* $columnToSortBy = Metrics::INDEX_NB_VISITS);
*
*
* $archiveProcessor->insertBlobRecords('MyPlugin_myFancyReport', $serializedData);
* }
*
*
* **Aggregating archive data**
*
*
* // function in Archiver descendant
* public function aggregateMultipleReports()
* {
* $archiveProcessor = $this->getProcessor();
*
*
* // aggregate a metric
* $archiveProcessor->aggregateNumericMetrics('MyPlugin_myFancyMetric');
* $archiveProcessor->aggregateNumericMetrics('MyPlugin_mySuperFancyMetric', 'max');
*
* // aggregate a report
*
* // aggregate a report
* $archiveProcessor->aggregateDataTableRecords('MyPlugin_myFancyReport');
* }
*
*
*/
class ArchiveProcessor
{
/**
* @var \Piwik\DataAccess\ArchiveWriter
*/
protected $archiveWriter;
private $archiveWriter;
/**
* @var \Piwik\DataAccess\LogAggregator
*/
protected $logAggregator;
private $logAggregator;
/**
* @var Archive
@ -94,28 +92,41 @@ class ArchiveProcessor
/**
* @var Parameters
*/
protected $params;
private $params;
/**
* @var int
*/
protected $numberOfVisits = false;
protected $numberOfVisitsConverted = false;
private $numberOfVisits = false;
public function __construct(Parameters $params, ArchiveWriter $archiveWriter)
private $numberOfVisitsConverted = false;
/**
* If true, unique visitors are not calculated when we are aggregating data for multiple sites.
* The `[General] enable_processing_unique_visitors_multiple_sites` INI config option controls
* the value of this variable.
*
* @var bool
*/
private $skipUniqueVisitorsCalculationForMultipleSites = true;
public function __construct(Parameters $params, ArchiveWriter $archiveWriter, LogAggregator $logAggregator)
{
$this->params = $params;
$this->logAggregator = new LogAggregator($params);
$this->logAggregator = $logAggregator;
$this->archiveWriter = $archiveWriter;
$this->skipUniqueVisitorsCalculationForMultipleSites = Rules::shouldSkipUniqueVisitorsCalculationForMultipleSites();
}
protected function getArchive()
{
if(empty($this->archive)) {
if (empty($this->archive)) {
$subPeriods = $this->params->getSubPeriods();
$idSites = $this->params->getIdSites();
$idSites = $this->params->getIdSites();
$this->archive = Archive::factory($this->params->getSegment(), $subPeriods, $idSites);
}
return $this->archive;
}
@ -155,7 +166,8 @@ class ArchiveProcessor
* @var array
*/
protected static $columnsToRenameAfterAggregation = array(
Metrics::INDEX_NB_UNIQ_VISITORS => Metrics::INDEX_SUM_DAILY_NB_UNIQ_VISITORS
Metrics::INDEX_NB_UNIQ_VISITORS => Metrics::INDEX_SUM_DAILY_NB_UNIQ_VISITORS,
Metrics::INDEX_NB_USERS => Metrics::INDEX_SUM_DAILY_NB_USERS,
);
/**
@ -172,8 +184,11 @@ class ArchiveProcessor
* @param array $columnsToRenameAfterAggregation Columns mapped to new names for columns that must change names
* when summed because they cannot be summed, eg,
* `array('nb_uniq_visitors' => 'sum_daily_nb_uniq_visitors')`.
* @param bool|array $countRowsRecursive if set to true, will calculate the recursive rows count for all record names
* which makes it slower. If you only need it for some records pass an array of
* recordNames that defines for which ones you need a recursive row count.
* @return array Returns the row counts of each aggregated report before truncation, eg,
*
*
* array(
* 'report1' => array('level0' => $report1->getRowsCount,
* 'recursive' => $report1->getRowsCountRecursive()),
@ -188,25 +203,23 @@ class ArchiveProcessor
$maximumRowsInSubDataTable = null,
$columnToSortByBeforeTruncation = null,
&$columnsAggregationOperation = null,
$columnsToRenameAfterAggregation = null)
$columnsToRenameAfterAggregation = null,
$countRowsRecursive = true)
{
if (!is_array($recordNames)) {
$recordNames = array($recordNames);
}
$nameToCount = array();
foreach ($recordNames as $recordName) {
$latestUsedTableId = Manager::getInstance()->getMostRecentTableId();
$table = $this->aggregateDataTableRecord($recordName, $columnsAggregationOperation, $columnsToRenameAfterAggregation);
$rowsCount = $table->getRowsCount();
$nameToCount[$recordName]['level0'] = $rowsCount;
$rowsCountRecursive = $rowsCount;
if($this->isAggregateSubTables()) {
$rowsCountRecursive = $table->getRowsCountRecursive();
$nameToCount[$recordName]['level0'] = $table->getRowsCount();
if ($countRowsRecursive === true || (is_array($countRowsRecursive) && in_array($recordName, $countRowsRecursive))) {
$nameToCount[$recordName]['recursive'] = $table->getRowsCountRecursive();
}
$nameToCount[$recordName]['recursive'] = $rowsCountRecursive;
$blob = $table->getSerialized($maximumRowsInDataTableLevelZero, $maximumRowsInSubDataTable, $columnToSortByBeforeTruncation);
Common::destroy($table);
@ -228,12 +241,12 @@ class ArchiveProcessor
* @return array|int Returns the array of aggregate values. If only one metric was aggregated,
* the aggregate value will be returned as is, not in an array.
* For example, if `array('nb_visits', 'nb_hits')` is supplied for `$columns`,
*
*
* array(
* 'nb_visits' => 3040,
* 'nb_hits' => 405
* )
*
*
* could be returned. If `array('nb_visits')` or `'nb_visits'` is used for `$columns`,
* then `3040` would be returned.
* @api
@ -242,7 +255,8 @@ class ArchiveProcessor
{
$metrics = $this->getAggregatedNumericMetrics($columns, $operationToApply);
foreach($metrics as $column => $value) {
foreach ($metrics as $column => $value) {
$value = Common::forceDotAsSeparatorForDecimalPoint($value);
$this->archiveWriter->insertRecord($column, $value);
}
// if asked for only one field to sum
@ -256,7 +270,7 @@ class ArchiveProcessor
public function getNumberOfVisits()
{
if($this->numberOfVisits === false) {
if ($this->numberOfVisits === false) {
throw new Exception("visits should have been set here");
}
return $this->numberOfVisits;
@ -273,7 +287,7 @@ class ArchiveProcessor
*
* @param array $numericRecords A name-value mapping of numeric values that should be
* archived, eg,
*
*
* array('Referrers_distinctKeywords' => 23, 'Referrers_distinctCampaigns' => 234)
* @api
*/
@ -297,6 +311,8 @@ class ArchiveProcessor
public function insertNumericRecord($name, $value)
{
$value = round($value, 2);
$value = Common::forceDotAsSeparatorForDecimalPoint($value);
$this->archiveWriter->insertRecord($name, $value);
}
@ -328,20 +344,53 @@ class ArchiveProcessor
*/
protected function aggregateDataTableRecord($name, $columnsAggregationOperation = null, $columnsToRenameAfterAggregation = null)
{
if($this->isAggregateSubTables()) {
// By default we shall aggregate all sub-tables.
$dataTable = $this->getArchive()->getDataTableExpanded($name, $idSubTable = null, $depth = null, $addMetadataSubtableId = false);
} else {
// In some cases (eg. Actions plugin when period=range),
// for better performance we will only aggregate the parent table
$dataTable = $this->getArchive()->getDataTable($name, $idSubTable = null);
// By default we shall aggregate all sub-tables.
$dataTable = $this->getArchive()->getDataTableExpanded($name, $idSubTable = null, $depth = null, $addMetadataSubtableId = false);
$columnsRenamed = false;
if ($dataTable instanceof Map) {
$columnsRenamed = true;
// see https://github.com/piwik/piwik/issues/4377
$self = $this;
$dataTable->filter(function ($table) use ($self, $columnsToRenameAfterAggregation) {
if ($self->areColumnsNotAlreadyRenamed($table)) {
/**
* This makes archiving and range dates a lot faster. Imagine we archive a week, then we will
* rename all columns of each 7 day archives. Afterwards we know the columns will be replaced in a
* week archive. When generating month archives, which uses mostly week archives, we do not have
* to replace those columns for the week archives again since we can be sure they were already
* replaced. Same when aggregating year and range archives. This can save up 10% or more when
* aggregating Month, Year and Range archives.
*/
$self->renameColumnsAfterAggregation($table, $columnsToRenameAfterAggregation);
}
});
}
$dataTable = $this->getAggregatedDataTableMap($dataTable, $columnsAggregationOperation);
$this->renameColumnsAfterAggregation($dataTable, $columnsToRenameAfterAggregation);
if (!$columnsRenamed) {
$this->renameColumnsAfterAggregation($dataTable, $columnsToRenameAfterAggregation);
}
return $dataTable;
}
/**
* Note: public only for use in closure in PHP 5.3.
*
* @param $table
* @return \Piwik\Period
*/
public function areColumnsNotAlreadyRenamed($table)
{
$period = $table->getMetadata(DataTableFactory::TABLE_METADATA_PERIOD_INDEX);
return !$period || $period->getLabel() === 'day';
}
protected function getOperationForColumns($columns, $defaultOperation)
{
$operationForColumn = array();
@ -357,18 +406,54 @@ class ArchiveProcessor
protected function enrichWithUniqueVisitorsMetric(Row $row)
{
if(!$this->getParams()->isSingleSite() ) {
// we only compute unique visitors for a single site
// skip unique visitors metrics calculation if calculating for multiple sites is disabled
if (!$this->getParams()->isSingleSite()
&& $this->skipUniqueVisitorsCalculationForMultipleSites
) {
return;
}
if ( $row->getColumn('nb_uniq_visitors') !== false) {
if (SettingsPiwik::isUniqueVisitorsEnabled($this->getParams()->getPeriod()->getLabel())) {
$uniqueVisitors = (float)$this->computeNbUniqVisitors();
$row->setColumn('nb_uniq_visitors', $uniqueVisitors);
} else {
$row->deleteColumn('nb_uniq_visitors');
}
if ($row->getColumn('nb_uniq_visitors') === false
&& $row->getColumn('nb_users') === false
) {
return;
}
if (!SettingsPiwik::isUniqueVisitorsEnabled($this->getParams()->getPeriod()->getLabel())) {
$row->deleteColumn('nb_uniq_visitors');
$row->deleteColumn('nb_users');
return;
}
$metrics = array(
Metrics::INDEX_NB_USERS
);
if ($this->getParams()->isSingleSite()) {
$uniqueVisitorsMetric = Metrics::INDEX_NB_UNIQ_VISITORS;
} else {
if (!SettingsPiwik::isSameFingerprintAcrossWebsites()) {
throw new Exception("Processing unique visitors across websites is enabled for this instance,
but to process this metric you must first set enable_fingerprinting_across_websites=1
in the config file, under the [Tracker] section.");
}
$uniqueVisitorsMetric = Metrics::INDEX_NB_UNIQ_FINGERPRINTS;
}
$metrics[] = $uniqueVisitorsMetric;
$uniques = $this->computeNbUniques($metrics);
// see edge case as described in https://github.com/piwik/piwik/issues/9357 where uniq_visitors might be higher
// than visits because we archive / process it after nb_visits. Between archiving nb_visits and nb_uniq_visitors
// there could have been a new visit leading to a higher nb_unique_visitors than nb_visits which is not possible
// by definition. In this case we simply use the visits metric instead of unique visitors metric.
$visits = $row->getColumn('nb_visits');
if ($visits !== false && $uniques[$uniqueVisitorsMetric] !== false) {
$uniques[$uniqueVisitorsMetric] = min($uniques[$uniqueVisitorsMetric], $visits);
}
$row->setColumn('nb_uniq_visitors', $uniques[$uniqueVisitorsMetric]);
$row->setColumn('nb_users', $uniques[Metrics::INDEX_NB_USERS]);
}
protected function guessOperationForColumn($column)
@ -388,14 +473,15 @@ class ArchiveProcessor
* This is the only Period metric (ie. week/month/year/range) that we process from the logs directly,
* since unique visitors cannot be summed like other metrics.
*
* @return int
* @param array Metrics Ids for which to aggregates count of values
* @return array of metrics, where the key is metricid and the value is the metric value
*/
protected function computeNbUniqVisitors()
protected function computeNbUniques($metrics)
{
$logAggregator = $this->getLogAggregator();
$query = $logAggregator->queryVisitsByDimension(array(), false, array(), array(Metrics::INDEX_NB_UNIQ_VISITORS));
$query = $logAggregator->queryVisitsByDimension(array(), false, array(), $metrics);
$data = $query->fetch();
return $data[Metrics::INDEX_NB_UNIQ_VISITORS];
return $data;
}
/**
@ -409,15 +495,18 @@ class ArchiveProcessor
protected function getAggregatedDataTableMap($data, $columnsAggregationOperation)
{
$table = new DataTable();
if (!empty($columnsAggregationOperation)) {
$table->setMetadata(DataTable::COLUMN_AGGREGATION_OPS_METADATA_NAME, $columnsAggregationOperation);
}
if ($data instanceof DataTable\Map) {
// as $date => $tableToSum
$this->aggregatedDataTableMapsAsOne($data, $table);
} else {
$table->addDataTable($data, $this->isAggregateSubTables());
$table->addDataTable($data);
}
return $table;
}
@ -429,22 +518,33 @@ class ArchiveProcessor
protected function aggregatedDataTableMapsAsOne(Map $map, DataTable $aggregated)
{
foreach ($map->getDataTables() as $tableToAggregate) {
if($tableToAggregate instanceof Map) {
if ($tableToAggregate instanceof Map) {
$this->aggregatedDataTableMapsAsOne($tableToAggregate, $aggregated);
} else {
$aggregated->addDataTable($tableToAggregate, $this->isAggregateSubTables());
$aggregated->addDataTable($tableToAggregate);
}
}
}
protected function renameColumnsAfterAggregation(DataTable $table, $columnsToRenameAfterAggregation = null)
/**
* Note: public only for use in closure in PHP 5.3.
*/
public function renameColumnsAfterAggregation(DataTable $table, $columnsToRenameAfterAggregation = null)
{
// Rename columns after aggregation
if (is_null($columnsToRenameAfterAggregation)) {
$columnsToRenameAfterAggregation = self::$columnsToRenameAfterAggregation;
}
foreach ($columnsToRenameAfterAggregation as $oldName => $newName) {
$table->renameColumn($oldName, $newName, $this->isAggregateSubTables());
foreach ($table->getRows() as $row) {
foreach ($columnsToRenameAfterAggregation as $oldName => $newName) {
$row->renameColumn($oldName, $newName);
}
$subTable = $row->getSubtable();
if ($subTable) {
$this->renameColumnsAfterAggregation($subTable, $columnsToRenameAfterAggregation);
}
}
}
@ -453,6 +553,7 @@ class ArchiveProcessor
if (!is_array($columns)) {
$columns = array($columns);
}
$operationForColumn = $this->getOperationForColumns($columns, $operationToApply);
$dataTable = $this->getArchive()->getDataTableFromNumeric($columns);
@ -463,11 +564,11 @@ class ArchiveProcessor
}
$rowMetrics = $results->getFirstRow();
if($rowMetrics === false) {
if ($rowMetrics === false) {
$rowMetrics = new Row;
}
$this->enrichWithUniqueVisitorsMetric($rowMetrics);
$this->renameColumnsAfterAggregation($results);
$this->renameColumnsAfterAggregation($results, self::$columnsToRenameAfterAggregation);
$metrics = $rowMetrics->getColumns();
@ -476,14 +577,7 @@ class ArchiveProcessor
$metrics[$name] = 0;
}
}
return $metrics;
}
/**
* @return bool
*/
protected function isAggregateSubTables()
{
return !$this->getParams()->isSkipAggregationOfSubTables();
}
}

57
www/analytics/core/ArchiveProcessor/Loader.php
View file

@ -1,18 +1,20 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\Archive;
use Piwik\ArchiveProcessor;
use Piwik\Cache;
use Piwik\Config;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\Date;
use Piwik\Period;
use Piwik\Piwik;
/**
* This class uses PluginsArchiver class to trigger data aggregation and create archives.
@ -81,6 +83,7 @@ class Loader
* Prepares the core metrics if needed.
*
* @param $visits
* @return array
*/
protected function prepareCoreMetricsArchive($visits, $visitsConverted)
{
@ -101,12 +104,14 @@ class Loader
$visits = $metrics['nb_visits'];
$visitsConverted = $metrics['nb_visits_converted'];
}
return array($visits, $visitsConverted);
}
protected function prepareAllPluginsArchive($visits, $visitsConverted)
{
$pluginsArchiver = new PluginsArchiver($this->params, $this->isArchiveTemporary());
if ($this->mustProcessVisitCount($visits)
|| $this->doesRequestedPluginIncludeVisitsSummary()
) {
@ -114,14 +119,14 @@ class Loader
$visits = $metrics['nb_visits'];
$visitsConverted = $metrics['nb_visits_converted'];
}
if ($this->isThereSomeVisits($visits)) {
if ($this->isThereSomeVisits($visits)
|| $this->shouldArchiveForSiteEvenWhenNoVisits()
) {
$pluginsArchiver->callAggregateAllPlugins($visits, $visitsConverted);
}
$idArchive = $pluginsArchiver->finalizeArchive();
if (!$this->params->isSingleSiteDayArchive() && $visits) {
ArchiveSelector::purgeOutdatedArchives($this->params->getPeriod()->getDateStart());
}
$idArchive = $pluginsArchiver->finalizeArchive();
return array($idArchive, $visits);
}
@ -139,11 +144,13 @@ class Loader
{
$period = $this->params->getPeriod()->getLabel();
$debugSetting = 'always_archive_data_period'; // default
if ($period == 'day') {
$debugSetting = 'always_archive_data_day';
} elseif ($period == 'range') {
$debugSetting = 'always_archive_data_range';
}
return (bool) Config::getInstance()->Debug[$debugSetting];
}
@ -165,9 +172,11 @@ class Loader
}
$idAndVisits = ArchiveSelector::getArchiveIdAndVisits($this->params, $minDatetimeArchiveProcessedUTC);
if (!$idAndVisits) {
return $noArchiveFound;
}
return $idAndVisits;
}
@ -186,19 +195,27 @@ class Loader
// Permanent archive
return $endDateTimestamp;
}
$dateStart = $this->params->getDateStart();
$period = $this->params->getPeriod();
$segment = $this->params->getSegment();
$site = $this->params->getSite();
// Temporary archive
return Rules::getMinTimeProcessedForTemporaryArchive($this->params->getDateStart(), $this->params->getPeriod(), $this->params->getSegment(), $this->params->getSite());
return Rules::getMinTimeProcessedForTemporaryArchive($dateStart, $period, $segment, $site);
}
protected static function determineIfArchivePermanent(Date $dateEnd)
{
$now = time();
$endTimestampUTC = strtotime($dateEnd->getDateEndUTC());
if ($endTimestampUTC <= $now) {
// - if the period we are looking for is finished, we look for a ts_archived that
// is greater than the last day of the archive
return $endTimestampUTC;
}
return false;
}
@ -207,8 +224,30 @@ class Loader
if (is_null($this->temporaryArchive)) {
throw new \Exception("getMinTimeArchiveProcessed() should be called prior to isArchiveTemporary()");
}
return $this->temporaryArchive;
}
}
private function shouldArchiveForSiteEvenWhenNoVisits()
{
$idSitesToArchive = $this->getIdSitesToArchiveWhenNoVisits();
return in_array($this->params->getSite()->getId(), $idSitesToArchive);
}
private function getIdSitesToArchiveWhenNoVisits()
{
$cache = Cache::getTransientCache();
$cacheKey = 'Archiving.getIdSitesToArchiveWhenNoVisits';
if (!$cache->contains($cacheKey)) {
$idSites = array();
// leaving undocumented unless decided otherwise
Piwik::postEvent('Archiving.getIdSitesToArchiveWhenNoVisits', array(&$idSites));
$cache->save($cacheKey, $idSites);
}
return $cache->fetch($cacheKey);
}
}

16
www/analytics/core/ArchiveProcessor/Parameters.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -45,15 +45,14 @@ class Parameters
/**
* Constructor.
*
*
* @ignore
*/
public function __construct(Site $site, Period $period, Segment $segment, $skipAggregationOfSubTables = false)
public function __construct(Site $site, Period $period, Segment $segment)
{
$this->site = $site;
$this->period = $period;
$this->segment = $segment;
$this->skipAggregationOfSubTables = $skipAggregationOfSubTables;
}
/**
@ -91,7 +90,7 @@ class Parameters
*/
public function getSubPeriods()
{
if($this->getPeriod()->getLabel() == 'day') {
if ($this->getPeriod()->getLabel() == 'day') {
return array( $this->getPeriod() );
}
return $this->getPeriod()->getSubperiods();
@ -169,18 +168,13 @@ class Parameters
return count($this->getIdSites()) == 1;
}
public function isSkipAggregationOfSubTables()
{
return $this->skipAggregationOfSubTables;
}
public function logStatusDebug($isTemporary)
{
$temporary = 'definitive archive';
if ($isTemporary) {
$temporary = 'temporary archive';
}
Log::verbose(
Log::debug(
"%s archive, idSite = %d (%s), segment '%s', report = '%s', UTC datetime [%s -> %s]",
$this->getPeriod()->getLabel(),
$this->getSite()->getId(),

74
www/analytics/core/ArchiveProcessor/PluginsArchiver.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,13 +9,15 @@
namespace Piwik\ArchiveProcessor;
use Piwik\Archive;
use Piwik\ArchiveProcessor;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\DataAccess\LogAggregator;
use Piwik\DataTable\Manager;
use Piwik\Metrics;
use Piwik\Plugin\Archiver;
use Piwik\Log;
use Piwik\Timer;
use Exception;
/**
* This class creates the Archiver objects found in plugins and will trigger aggregation,
@ -34,9 +36,16 @@ class PluginsArchiver
protected $params;
/**
* @var LogAggregator
*/
private $logAggregator;
/**
* Public only for tests. Won't be necessary after DI changes are complete.
*
* @var Archiver[] $archivers
*/
private static $archivers = array();
public static $archivers = array();
public function __construct(Parameters $params, $isTemporaryArchive)
{
@ -45,7 +54,9 @@ class PluginsArchiver
$this->archiveWriter = new ArchiveWriter($this->params, $isTemporaryArchive);
$this->archiveWriter->initNewArchive();
$this->archiveProcessor = new ArchiveProcessor($this->params, $this->archiveWriter);
$this->logAggregator = new LogAggregator($params);
$this->archiveProcessor = new ArchiveProcessor($this->params, $this->archiveWriter, $this->logAggregator);
$this->isSingleSiteDayArchive = $this->params->isSingleSiteDayArchive();
}
@ -57,7 +68,9 @@ class PluginsArchiver
*/
public function callAggregateCoreMetrics()
{
if($this->isSingleSiteDayArchive) {
$this->logAggregator->setQueryOriginHint('Core');
if ($this->isSingleSiteDayArchive) {
$metrics = $this->aggregateDayVisitsMetrics();
} else {
$metrics = $this->aggregateMultipleVisitsMetrics();
@ -81,27 +94,57 @@ class PluginsArchiver
*/
public function callAggregateAllPlugins($visits, $visitsConverted)
{
Log::debug("PluginsArchiver::%s: Initializing archiving process for all plugins [visits = %s, visits converted = %s]",
__FUNCTION__, $visits, $visitsConverted);
$this->archiveProcessor->setNumberOfVisits($visits, $visitsConverted);
$archivers = $this->getPluginArchivers();
foreach($archivers as $pluginName => $archiverClass) {
foreach ($archivers as $pluginName => $archiverClass) {
// We clean up below all tables created during this function call (and recursive calls)
$latestUsedTableId = Manager::getInstance()->getMostRecentTableId();
/** @var Archiver $archiver */
$archiver = new $archiverClass($this->archiveProcessor);
if(!$archiver->isEnabled()) {
if (!$archiver->isEnabled()) {
Log::debug("PluginsArchiver::%s: Skipping archiving for plugin '%s'.", __FUNCTION__, $pluginName);
continue;
}
if($this->shouldProcessReportsForPlugin($pluginName)) {
if($this->isSingleSiteDayArchive) {
$archiver->aggregateDayReport();
} else {
$archiver->aggregateMultipleReports();
if ($this->shouldProcessReportsForPlugin($pluginName)) {
$this->logAggregator->setQueryOriginHint($pluginName);
try {
$timer = new Timer();
if ($this->isSingleSiteDayArchive) {
Log::debug("PluginsArchiver::%s: Archiving day reports for plugin '%s'.", __FUNCTION__, $pluginName);
$archiver->aggregateDayReport();
} else {
Log::debug("PluginsArchiver::%s: Archiving period reports for plugin '%s'.", __FUNCTION__, $pluginName);
$archiver->aggregateMultipleReports();
}
$this->logAggregator->setQueryOriginHint('');
Log::debug("PluginsArchiver::%s: %s while archiving %s reports for plugin '%s'.",
__FUNCTION__,
$timer->getMemoryLeak(),
$this->params->getPeriod()->getLabel(),
$pluginName
);
} catch (Exception $e) {
$className = get_class($e);
$exception = new $className($e->getMessage() . " - caused by plugin $pluginName", $e->getCode(), $e);
throw $exception;
}
} else {
Log::debug("PluginsArchiver::%s: Not archiving reports for plugin '%s'.", __FUNCTION__, $pluginName);
}
Manager::getInstance()->deleteAll($latestUsedTableId);
@ -111,7 +154,7 @@ class PluginsArchiver
public function finalizeArchive()
{
$this->params->logStatusDebug( $this->archiveWriter->isArchiveTemporary );
$this->params->logStatusDebug($this->archiveWriter->isArchiveTemporary);
$this->archiveWriter->finalizeArchive();
return $this->archiveWriter->getIdArchive();
}
@ -193,5 +236,4 @@ class PluginsArchiver
$metrics = $this->archiveProcessor->aggregateNumericMetrics($toSum);
return $metrics;
}
}

156
www/analytics/core/ArchiveProcessor/Rules.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,14 +9,13 @@
namespace Piwik\ArchiveProcessor;
use Exception;
use Piwik\Common;
use Piwik\Config;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\Date;
use Piwik\Log;
use Piwik\Option;
use Piwik\Piwik;
use Piwik\Plugins\CoreAdminHome\Controller;
use Piwik\Plugins\CoreAdminHome\CoreAdminHome;
use Piwik\Segment;
use Piwik\SettingsPiwik;
use Piwik\SettingsServer;
@ -25,7 +24,7 @@ use Piwik\Tracker\Cache;
/**
* This class contains Archiving rules/logic which are used when creating and processing Archives.
*
*
*/
class Rules
{
@ -35,9 +34,6 @@ class Rules
const FLAG_TABLE_PURGED = 'lastPurge_';
/** Old Archives purge can be disabled (used in tests only) */
static public $purgeDisabledByTests = false;
/** Flag that will forcefully disable the archiving process (used in tests only) */
public static $archivingDisabledByTests = false;
@ -45,15 +41,16 @@ class Rules
* Returns the name of the archive field used to tell the status of an archive, (ie,
* whether the archive was created successfully or not).
*
* @param array $idSites
* @param Segment $segment
* @param string $periodLabel
* @param string $plugin
* @return string
*/
public static function getDoneStringFlagFor(array $idSites, $segment, $periodLabel, $plugin, $isSkipAggregationOfSubTables)
public static function getDoneStringFlagFor(array $idSites, $segment, $periodLabel, $plugin)
{
if (!self::shouldProcessReportsAllPlugins($idSites, $segment, $periodLabel)) {
return self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin, $isSkipAggregationOfSubTables);
return self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin);
}
return self::getDoneFlagArchiveContainsAllPlugins($segment);
}
@ -84,39 +81,24 @@ class Rules
return $segmentsToProcess;
}
public static function getDoneFlagArchiveContainsOnePlugin(Segment $segment, $plugin, $isSkipAggregationOfSubTables = false)
public static function getDoneFlagArchiveContainsOnePlugin(Segment $segment, $plugin)
{
$partial = self::isFlagArchivePartial($plugin, $isSkipAggregationOfSubTables);
return 'done' . $segment->getHash() . '.' . $plugin . $partial ;
return 'done' . $segment->getHash() . '.' . $plugin ;
}
private static function getDoneFlagArchiveContainsAllPlugins(Segment $segment)
public static function getDoneFlagArchiveContainsAllPlugins(Segment $segment)
{
return 'done' . $segment->getHash();
}
/**
* @param $plugin
* @param $isSkipAggregationOfSubTables
* @return string
*/
private static function isFlagArchivePartial($plugin, $isSkipAggregationOfSubTables)
{
$partialArchive = '';
if ($plugin != "VisitsSummary" // VisitsSummary is always called when segmenting and should not have its own .partial archive
&& $isSkipAggregationOfSubTables
) {
$partialArchive = '.partial';
}
return $partialArchive;
}
/**
* Return done flags used to tell how the archiving process for a specific archive was completed,
*
* @param array $plugins
* @param $segment
* @return array
*/
public static function getDoneFlags(array $plugins, Segment $segment, $isSkipAggregationOfSubTables)
public static function getDoneFlags(array $plugins, Segment $segment)
{
$doneFlags = array();
$doneAllPlugins = self::getDoneFlagArchiveContainsAllPlugins($segment);
@ -124,58 +106,12 @@ class Rules
$plugins = array_unique($plugins);
foreach ($plugins as $plugin) {
$doneOnePlugin = self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin, $isSkipAggregationOfSubTables);
$doneOnePlugin = self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin);
$doneFlags[$plugin] = $doneOnePlugin;
}
return $doneFlags;
}
/**
* Given a monthly archive table, will delete all reports that are now outdated,
* or reports that ended with an error
*
* @param \Piwik\Date $date
* @return int|bool False, or timestamp indicating which archives to delete
*/
public static function shouldPurgeOutdatedArchives(Date $date)
{
if (self::$purgeDisabledByTests) {
return false;
}
$key = self::FLAG_TABLE_PURGED . "blob_" . $date->toString('Y_m');
$timestamp = Option::get($key);
// we shall purge temporary archives after their timeout is finished, plus an extra 6 hours
// in case archiving is disabled or run once a day, we give it this extra time to run
// and re-process more recent records...
$temporaryArchivingTimeout = self::getTodayArchiveTimeToLive();
$hoursBetweenPurge = 6;
$purgeEveryNSeconds = max($temporaryArchivingTimeout, $hoursBetweenPurge * 3600);
// we only delete archives if we are able to process them, otherwise, the browser might process reports
// when &segment= is specified (or custom date range) and would below, delete temporary archives that the
// browser is not able to process until next cron run (which could be more than 1 hour away)
if (self::isRequestAuthorizedToArchive()
&& (!$timestamp
|| $timestamp < time() - $purgeEveryNSeconds)
) {
Option::set($key, time());
if (self::isBrowserTriggerEnabled()) {
// If Browser Archiving is enabled, it is likely there are many more temporary archives
// We delete more often which is safe, since reports are re-processed on demand
$purgeArchivesOlderThan = Date::factory(time() - 2 * $temporaryArchivingTimeout)->getDateTime();
} else {
// If archive.php via Cron is building the reports, we should keep all temporary reports from today
$purgeArchivesOlderThan = Date::factory('today')->getDateTime();
}
return $purgeArchivesOlderThan;
}
Log::info("Purging temporary archives: skipped.");
return false;
}
public static function getMinTimeProcessedForTemporaryArchive(
Date $dateStart, \Piwik\Period $period, Segment $segment, Site $site)
{
@ -213,30 +149,45 @@ class Rules
{
$uiSettingIsEnabled = Controller::isGeneralSettingsAdminEnabled();
if($uiSettingIsEnabled) {
if ($uiSettingIsEnabled) {
$timeToLive = Option::get(self::OPTION_TODAY_ARCHIVE_TTL);
if ($timeToLive !== false) {
return $timeToLive;
}
}
return self::getTodayArchiveTimeToLiveDefault();
}
public static function getTodayArchiveTimeToLiveDefault()
{
return Config::getInstance()->General['time_before_today_archive_considered_outdated'];
}
public static function isArchivingDisabledFor(array $idSites, Segment $segment, $periodLabel)
{
$generalConfig = Config::getInstance()->General;
if ($periodLabel == 'range') {
return false;
if (!isset($generalConfig['archiving_range_force_on_browser_request'])
|| $generalConfig['archiving_range_force_on_browser_request'] != false
) {
return false;
} else {
Log::debug("Not forcing archiving for range period.");
}
}
$processOneReportOnly = !self::shouldProcessReportsAllPlugins($idSites, $segment, $periodLabel);
$isArchivingDisabled = !self::isRequestAuthorizedToArchive() || self::$archivingDisabledByTests;
if ($processOneReportOnly) {
if ($processOneReportOnly
&& $periodLabel != 'range'
) {
// When there is a segment, we disable archiving when browser_archiving_disabled_enforce applies
if (!$segment->isEmpty()
&& $isArchivingDisabled
&& Config::getInstance()->General['browser_archiving_disabled_enforce']
&& !SettingsServer::isArchivePhpTriggered() // Only applies when we are not running archive.php
&& $generalConfig['browser_archiving_disabled_enforce']
&& !SettingsServer::isArchivePhpTriggered() // Only applies when we are not running core:archive command
) {
Log::debug("Archiving is disabled because of config setting browser_archiving_disabled_enforce=1");
return true;
@ -248,7 +199,7 @@ class Rules
return $isArchivingDisabled;
}
protected static function isRequestAuthorizedToArchive()
public static function isRequestAuthorizedToArchive()
{
return Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();
}
@ -257,7 +208,7 @@ class Rules
{
$uiSettingIsEnabled = Controller::isGeneralSettingsAdminEnabled();
if($uiSettingIsEnabled) {
if ($uiSettingIsEnabled) {
$browserArchivingEnabled = Option::get(self::OPTION_BROWSER_TRIGGER_ARCHIVING);
if ($browserArchivingEnabled !== false) {
return (bool)$browserArchivingEnabled;
@ -275,6 +226,18 @@ class Rules
Cache::clearCacheGeneral();
}
/**
* Returns true if the archiving process should skip the calculation of unique visitors
* across several sites. The `[General] enable_processing_unique_visitors_multiple_sites`
* INI config option controls the value of this variable.
*
* @return bool
*/
public static function shouldSkipUniqueVisitorsCalculationForMultipleSites()
{
return Config::getInstance()->General['enable_processing_unique_visitors_multiple_sites'] != 1;
}
/**
* @param array $idSites
* @param Segment $segment
@ -294,11 +257,24 @@ class Rules
// Turns out the getString() above returns the URL decoded segment string
$segmentsToProcessUrlDecoded = array_map('urldecode', $segmentsToProcess);
if (in_array($segment, $segmentsToProcess)
|| in_array($segment, $segmentsToProcessUrlDecoded)
) {
return true;
return in_array($segment, $segmentsToProcess)
|| in_array($segment, $segmentsToProcessUrlDecoded);
}
/**
* Returns done flag values allowed to be selected
*
* @return string
*/
public static function getSelectableDoneFlagValues()
{
$possibleValues = array(ArchiveWriter::DONE_OK, ArchiveWriter::DONE_OK_TEMPORARY);
if (!Rules::isRequestAuthorizedToArchive()) {
//If request is not authorized to archive then fetch also invalidated archives
$possibleValues[] = ArchiveWriter::DONE_INVALIDATED;
}
return false;
return $possibleValues;
}
}

48
www/analytics/core/Archiver/Request.php Normal file
View file

@ -0,0 +1,48 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archiver;
class Request
{
/**
* @var string
*/
private $url;
/**
* @var callable|null
*/
private $before;
/**
* @param string $url
*/
public function __construct($url)
{
$this->url = $url;
}
public function before($callable)
{
$this->before = $callable;
}
public function start()
{
if ($this->before) {
$callable = $this->before;
$callable();
}
}
public function __toString()
{
return $this->url;
}
}

64
www/analytics/core/AssetManager.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -19,9 +19,8 @@ use Piwik\AssetManager\UIAssetFetcher\StylesheetUIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetMerger\JScriptUIAssetMerger;
use Piwik\AssetManager\UIAssetMerger\StylesheetUIAssetMerger;
use Piwik\Container\StaticContainer;
use Piwik\Plugin\Manager;
use Piwik\Translate;
use Piwik\Config as PiwikConfig;
/**
* AssetManager is the class used to manage the inclusion of UI assets:
@ -37,7 +36,7 @@ use Piwik\Config as PiwikConfig;
* the global option 'disable_merged_assets'. See the documentation in the global
* config for more information.
*
* @method static \Piwik\AssetManager getInstance()
* @method static AssetManager getInstance()
*/
class AssetManager extends Singleton
{
@ -66,13 +65,13 @@ class AssetManager extends Singleton
*/
private $theme;
function __construct()
public function __construct()
{
$this->cacheBuster = UIAssetCacheBuster::getInstance();
$this->minimalStylesheetFetcher = new StaticUIAssetFetcher(array('plugins/Zeitgeist/stylesheets/base.less'), array(), $this->theme);
$this->minimalStylesheetFetcher = new StaticUIAssetFetcher(array('plugins/Morpheus/stylesheets/base.less', 'plugins/Morpheus/stylesheets/general/_forms.less'), array(), $this->theme);
$theme = Manager::getInstance()->getThemeEnabled();
if(!empty($theme)) {
if (!empty($theme)) {
$this->theme = new Theme();
}
}
@ -121,14 +120,11 @@ class AssetManager extends Singleton
$result = "<script type=\"text/javascript\">\n" . Translate::getJavascriptTranslations() . "\n</script>";
if ($this->isMergedAssetsDisabled()) {
$this->getMergedCoreJSAsset()->delete();
$this->getMergedNonCoreJSAsset()->delete();
$result .= $this->getIndividualJsIncludes();
} else {
$result .= sprintf(self::JS_IMPORT_DIRECTIVE, self::GET_CORE_JS_MODULE_ACTION);
$result .= sprintf(self::JS_IMPORT_DIRECTIVE, self::GET_NON_CORE_JS_MODULE_ACTION);
}
@ -201,13 +197,13 @@ class AssetManager extends Singleton
{
$loadedPlugins = array();
foreach(Manager::getInstance()->getPluginsLoadedAndActivated() as $plugin) {
foreach (Manager::getInstance()->getPluginsLoadedAndActivated() as $plugin) {
$pluginName = $plugin->getPluginName();
$pluginIsCore = Manager::getInstance()->isPluginBundledWithCore($pluginName);
if(($pluginIsCore && $core) || (!$pluginIsCore && !$core))
if (($pluginIsCore && $core) || (!$pluginIsCore && !$core)) {
$loadedPlugins[] = $pluginName;
}
}
return $loadedPlugins;
@ -220,23 +216,15 @@ class AssetManager extends Singleton
{
$assetsToRemove = array($this->getMergedStylesheetAsset());
if($pluginName) {
if($this->pluginContainsJScriptAssets($pluginName)) {
PiwikConfig::getInstance()->init();
if(Manager::getInstance()->isPluginBundledWithCore($pluginName)) {
if ($pluginName) {
if ($this->pluginContainsJScriptAssets($pluginName)) {
if (Manager::getInstance()->isPluginBundledWithCore($pluginName)) {
$assetsToRemove[] = $this->getMergedCoreJSAsset();
} else {
$assetsToRemove[] = $this->getMergedNonCoreJSAsset();
}
}
} else {
$assetsToRemove[] = $this->getMergedCoreJSAsset();
$assetsToRemove[] = $this->getMergedNonCoreJSAsset();
}
@ -252,8 +240,7 @@ class AssetManager extends Singleton
*/
public function getAssetDirectory()
{
$mergedFileDirectory = PIWIK_USER_PATH . "/tmp/assets";
$mergedFileDirectory = SettingsPiwik::rewriteTmpPathWithHostname($mergedFileDirectory);
$mergedFileDirectory = StaticContainer::get('path.tmp') . '/assets';
if (!is_dir($mergedFileDirectory)) {
Filesystem::mkdir($mergedFileDirectory);
@ -273,7 +260,15 @@ class AssetManager extends Singleton
*/
public function isMergedAssetsDisabled()
{
return Config::getInstance()->Debug['disable_merged_assets'];
if (Config::getInstance()->Development['disable_merged_assets'] == 1) {
return true;
}
if (isset($_GET['disable_merged_assets']) && $_GET['disable_merged_assets'] == 1) {
return true;
}
return false;
}
/**
@ -311,7 +306,6 @@ class AssetManager extends Singleton
$jsIncludeString = '';
foreach ($assetFetcher->getCatalog()->getAssets() as $jsFile) {
$jsFile->validateFile();
$jsIncludeString = $jsIncludeString . sprintf(self::JS_IMPORT_DIRECTIVE, $jsFile->getRelativeLocation());
}
@ -339,7 +333,7 @@ class AssetManager extends Singleton
try {
$assets = $fetcher->getCatalog()->getAssets();
} catch(\Exception $e) {
} catch (\Exception $e) {
// This can happen when a plugin is not valid (eg. Piwik 1.x format)
// When posting the event to the plugin, it returns an exception "Plugin has not been loaded"
return false;
@ -347,14 +341,14 @@ class AssetManager extends Singleton
$plugin = Manager::getInstance()->getLoadedPlugin($pluginName);
if($plugin->isTheme()) {
if ($plugin->isTheme()) {
$theme = Manager::getInstance()->getTheme($pluginName);
$javaScriptFiles = $theme->getJavaScriptFiles();
if(!empty($javaScriptFiles))
if (!empty($javaScriptFiles)) {
$assets = array_merge($assets, $javaScriptFiles);
}
}
return !empty($assets);
@ -363,9 +357,9 @@ class AssetManager extends Singleton
/**
* @param UIAsset[] $uiAssets
*/
private function removeAssets($uiAssets)
public function removeAssets($uiAssets)
{
foreach($uiAssets as $uiAsset) {
foreach ($uiAssets as $uiAsset) {
$uiAsset->delete();
}
}
@ -373,7 +367,7 @@ class AssetManager extends Singleton
/**
* @return UIAsset
*/
private function getMergedStylesheetAsset()
public function getMergedStylesheetAsset()
{
return $this->getMergedUIAsset(self::MERGED_CSS_FILE);
}

2
www/analytics/core/AssetManager/UIAsset.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later

3
www/analytics/core/AssetManager/UIAsset/InMemoryUIAsset.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -45,7 +45,6 @@ class InMemoryUIAsset extends UIAsset
return false;
}
public function writeContent($content)
{
$this->content = $content;

23
www/analytics/core/AssetManager/UIAsset/OnDiskUIAsset.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,6 +10,7 @@ namespace Piwik\AssetManager\UIAsset;
use Exception;
use Piwik\AssetManager\UIAsset;
use Piwik\Filesystem;
class OnDiskUIAsset extends UIAsset
{
@ -27,7 +28,7 @@ class OnDiskUIAsset extends UIAsset
* @param string $baseDirectory
* @param string $fileLocation
*/
function __construct($baseDirectory, $fileLocation)
public function __construct($baseDirectory, $fileLocation)
{
$this->baseDirectory = $baseDirectory;
$this->relativeLocation = $fileLocation;
@ -50,20 +51,23 @@ class OnDiskUIAsset extends UIAsset
public function validateFile()
{
if (!$this->assetIsReadable())
if (!$this->assetIsReadable()) {
throw new Exception("The ui asset with 'href' = " . $this->getAbsoluteLocation() . " is not readable");
}
}
public function delete()
{
if ($this->exists()) {
if (!unlink($this->getAbsoluteLocation()))
try {
Filesystem::remove($this->getAbsoluteLocation());
} catch (Exception $e) {
throw new Exception("Unable to delete merged file : " . $this->getAbsoluteLocation() . ". Please delete the file and refresh");
}
// try to remove compressed version of the merged file.
@unlink($this->getAbsoluteLocation() . ".deflate");
@unlink($this->getAbsoluteLocation() . ".gz");
Filesystem::remove($this->getAbsoluteLocation() . ".deflate", true);
Filesystem::remove($this->getAbsoluteLocation() . ".gz", true);
}
}
@ -77,8 +81,9 @@ class OnDiskUIAsset extends UIAsset
$newFile = @fopen($this->getAbsoluteLocation(), "w");
if (!$newFile)
throw new Exception ("The file : " . $newFile . " can not be opened in write mode.");
if (!$newFile) {
throw new Exception("The file : " . $newFile . " can not be opened in write mode.");
}
fwrite($newFile, $content);

2
www/analytics/core/AssetManager/UIAssetCacheBuster.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later

21
www/analytics/core/AssetManager/UIAssetCatalog.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -20,10 +20,15 @@ class UIAssetCatalog
*/
private $catalogSorter;
/**
* @var string[] Absolute file locations
*/
private $existingAssetLocations = array();
/**
* @param UIAssetCatalogSorter $catalogSorter
*/
function __construct($catalogSorter)
public function __construct($catalogSorter)
{
$this->catalogSorter = $catalogSorter;
}
@ -33,8 +38,10 @@ class UIAssetCatalog
*/
public function addUIAsset($uiAsset)
{
if(!$this->assetAlreadyInCatalog($uiAsset)) {
$location = $uiAsset->getAbsoluteLocation();
if (!$this->assetAlreadyInCatalog($location)) {
$this->existingAssetLocations[] = $location;
$this->uiAssets[] = $uiAsset;
}
}
@ -59,12 +66,8 @@ class UIAssetCatalog
* @param UIAsset $uiAsset
* @return boolean
*/
private function assetAlreadyInCatalog($uiAsset)
private function assetAlreadyInCatalog($location)
{
foreach($this->uiAssets as $existingAsset)
if($uiAsset->getAbsoluteLocation() == $existingAsset->getAbsoluteLocation())
return true;
return false;
return in_array($location, $this->existingAssetLocations);
}
}

9
www/analytics/core/AssetManager/UIAssetCatalogSorter.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -18,7 +18,7 @@ class UIAssetCatalogSorter
/**
* @param string[] $priorityOrder
*/
function __construct($priorityOrder)
public function __construct($priorityOrder)
{
$this->priorityOrder = $priorityOrder;
}
@ -31,12 +31,11 @@ class UIAssetCatalogSorter
{
$sortedCatalog = new UIAssetCatalog($this);
foreach ($this->priorityOrder as $filePattern) {
$assetsMatchingPattern = array_filter($uiAssetCatalog->getAssets(), function($uiAsset) use ($filePattern) {
$assetsMatchingPattern = array_filter($uiAssetCatalog->getAssets(), function ($uiAsset) use ($filePattern) {
return preg_match('~^' . $filePattern . '~', $uiAsset->getRelativeLocation());
});
foreach($assetsMatchingPattern as $assetMatchingPattern) {
foreach ($assetsMatchingPattern as $assetMatchingPattern) {
$sortedCatalog->addUIAsset($assetMatchingPattern);
}
}

8
www/analytics/core/AssetManager/UIAssetFetcher.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -37,7 +37,7 @@ abstract class UIAssetFetcher
* @param string[] $plugins
* @param Theme $theme
*/
function __construct($plugins, $theme)
public function __construct($plugins, $theme)
{
$this->plugins = $plugins;
$this->theme = $theme;
@ -56,8 +56,9 @@ abstract class UIAssetFetcher
*/
public function getCatalog()
{
if($this->catalog == null)
if ($this->catalog == null) {
$this->createCatalog();
}
return $this->catalog;
}
@ -89,7 +90,6 @@ abstract class UIAssetFetcher
private function populateCatalog()
{
foreach ($this->fileLocations as $fileLocation) {
$newUIAsset = new OnDiskUIAsset($this->getBaseDirectory(), $fileLocation);
$this->catalog->addUIAsset($newUIAsset);
}

31
www/analytics/core/AssetManager/UIAssetFetcher/JScriptUIAssetFetcher.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,15 +10,13 @@ namespace Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\Piwik;
use string;
class JScriptUIAssetFetcher extends UIAssetFetcher
{
protected function retrieveFileLocations()
{
if(!empty($this->plugins)) {
if (!empty($this->plugins)) {
/**
* Triggered when gathering the list of all JavaScript files needed by Piwik
@ -31,7 +29,7 @@ class JScriptUIAssetFetcher extends UIAssetFetcher
* plugin's root directory.
*
* _Note: While you are developing your plugin you should enable the config setting
* `[Debug] disable_merged_assets` so JavaScript files will be reloaded immediately
* `[Development] disable_merged_assets` so JavaScript files will be reloaded immediately
* after every change._
*
* **Example**
@ -53,17 +51,14 @@ class JScriptUIAssetFetcher extends UIAssetFetcher
protected function addThemeFiles()
{
$theme = $this->getTheme();
if(!$theme) {
if (!$theme) {
return;
}
if(in_array($theme->getThemeName(), $this->plugins)) {
if (in_array($theme->getThemeName(), $this->plugins)) {
$jsInThemes = $this->getTheme()->getJavaScriptFiles();
if(!empty($jsInThemes)) {
foreach($jsInThemes as $jsFile) {
if (!empty($jsInThemes)) {
foreach ($jsInThemes as $jsFile) {
$this->fileLocations[] = $jsFile;
}
}
@ -73,13 +68,17 @@ class JScriptUIAssetFetcher extends UIAssetFetcher
protected function getPriorityOrder()
{
return array(
'libs/jquery/jquery.js',
'libs/jquery/jquery-ui.js',
'libs/bower_components/jquery/dist/jquery.min.js',
'libs/bower_components/jquery-ui/ui/minified/jquery-ui.min.js',
'libs/jquery/jquery.browser.js',
'libs/',
'js/',
'piwik.js',
'plugins/CoreHome/javascripts/require.js',
'plugins/Zeitgeist/javascripts/piwikHelper.js',
'plugins/Zeitgeist/javascripts/',
'plugins/Morpheus/javascripts/piwikHelper.js',
'plugins/Morpheus/javascripts/jquery.icheck.min.js',
'plugins/Morpheus/javascripts/morpheus.js',
'plugins/Morpheus/javascripts/',
'plugins/CoreHome/javascripts/uiControl.js',
'plugins/CoreHome/javascripts/broadcast.js',
'plugins/CoreHome/javascripts/', // load CoreHome JS before other plugins

5
www/analytics/core/AssetManager/UIAssetFetcher/StaticUIAssetFetcher.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -17,7 +17,7 @@ class StaticUIAssetFetcher extends UIAssetFetcher
*/
private $priorityOrder;
function __construct($fileLocations, $priorityOrder, $theme)
public function __construct($fileLocations, $priorityOrder, $theme)
{
parent::__construct(array(), $theme);
@ -27,7 +27,6 @@ class StaticUIAssetFetcher extends UIAssetFetcher
protected function retrieveFileLocations()
{
}
protected function getPriorityOrder()

35
www/analytics/core/AssetManager/UIAssetFetcher/StylesheetUIAssetFetcher.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -15,15 +15,30 @@ class StylesheetUIAssetFetcher extends UIAssetFetcher
{
protected function getPriorityOrder()
{
return array(
$theme = $this->getTheme();
$themeName = $theme->getThemeName();
$order = array(
'libs/',
'plugins/CoreHome/stylesheets/color_manager.css', // must be before other Piwik stylesheets
'plugins/Zeitgeist/stylesheets/base.less',
'plugins/Zeitgeist/stylesheets/',
'plugins/',
'plugins/Dashboard/stylesheets/dashboard.less',
'tests/',
'plugins/Morpheus/stylesheets/base.less',
);
if ($themeName === 'Morpheus') {
$order[] = 'plugins\/((?!Morpheus).)*\/';
} else {
$order[] = sprintf('plugins\/((?!(Morpheus)|(%s)).)*\/', $themeName);
}
$order = array_merge(
$order,
array(
'plugins/Dashboard/stylesheets/dashboard.less',
'tests/',
)
);
return $order;
}
protected function retrieveFileLocations()
@ -55,9 +70,13 @@ class StylesheetUIAssetFetcher extends UIAssetFetcher
protected function addThemeFiles()
{
$theme = $this->getTheme();
if (!$theme) {
return;
}
$themeStylesheet = $this->getTheme()->getStylesheet();
if($themeStylesheet) {
if ($themeStylesheet) {
$this->fileLocations[] = $themeStylesheet;
}
}

38
www/analytics/core/AssetManager/UIAssetMerger.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,9 +8,6 @@
*/
namespace Piwik\AssetManager;
use Piwik\AssetManager\PiwikLessCompiler;
use Piwik\AssetManager\UIAsset\StylesheetUIAsset;
use Piwik\AssetManager;
abstract class UIAssetMerger
{
@ -39,7 +36,7 @@ abstract class UIAssetMerger
* @param UIAssetFetcher $assetFetcher
* @param UIAssetCacheBuster $cacheBuster
*/
function __construct($mergedAsset, $assetFetcher, $cacheBuster)
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
$this->mergedAsset = $mergedAsset;
$this->assetFetcher = $assetFetcher;
@ -48,8 +45,9 @@ abstract class UIAssetMerger
public function generateFile()
{
if(!$this->shouldGenerate())
if (!$this->shouldGenerate()) {
return;
}
$this->mergedContent = $this->getMergedAssets();
@ -95,8 +93,9 @@ abstract class UIAssetMerger
protected function getConcatenatedAssets()
{
if(empty($this->mergedContent))
if (empty($this->mergedContent)) {
$this->concatenateAssets();
}
return $this->mergedContent;
}
@ -106,7 +105,6 @@ abstract class UIAssetMerger
$mergedContent = '';
foreach ($this->getAssetCatalog()->getAssets() as $uiAsset) {
$uiAsset->validateFile();
$content = $this->processFileContent($uiAsset);
@ -137,8 +135,9 @@ abstract class UIAssetMerger
*/
private function shouldGenerate()
{
if(!$this->mergedAsset->exists())
if (!$this->mergedAsset->exists()) {
return true;
}
return !$this->isFileUpToDate();
}
@ -161,19 +160,11 @@ abstract class UIAssetMerger
return false;
}
/**
* @return boolean
*/
private function isMergedAssetsDisabled()
{
return AssetManager::getInstance()->isMergedAssetsDisabled();
}
private function adjustPaths()
{
$theme = $this->assetFetcher->getTheme();
// During installation theme is not yet ready
if($theme) {
if ($theme) {
$this->mergedContent = $this->assetFetcher->getTheme()->rewriteAssetsPathToTheme($this->mergedContent);
}
}
@ -188,8 +179,9 @@ abstract class UIAssetMerger
*/
protected function getCacheBusterValue()
{
if(empty($this->cacheBusterValue))
if (empty($this->cacheBusterValue)) {
$this->cacheBusterValue = $this->generateCacheBuster();
}
return $this->cacheBusterValue;
}
@ -198,12 +190,4 @@ abstract class UIAssetMerger
{
$this->mergedContent = $this->getPreamble() . $this->mergedContent;
}
/**
* @return boolean
*/
private function shouldCompareExistingVersion()
{
return $this->isMergedAssetsDisabled();
}
}

17
www/analytics/core/AssetManager/UIAssetMerger/JScriptUIAssetMerger.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -28,7 +28,7 @@ class JScriptUIAssetMerger extends UIAssetMerger
* @param JScriptUIAssetFetcher $assetFetcher
* @param UIAssetCacheBuster $cacheBuster
*/
function __construct($mergedAsset, $assetFetcher, $cacheBuster)
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
parent::__construct($mergedAsset, $assetFetcher, $cacheBuster);
@ -37,15 +37,13 @@ class JScriptUIAssetMerger extends UIAssetMerger
protected function getMergedAssets()
{
$concatenatedAssets = $this->getConcatenatedAssets();
return str_replace("\n", "\r\n", $concatenatedAssets);
return $this->getConcatenatedAssets();
}
protected function generateCacheBuster()
{
$cacheBuster = $this->cacheBuster->piwikVersionBasedCacheBuster($this->getPlugins());
return "/* Piwik Javascript - cb=" . $cacheBuster . "*/\r\n";
return "/* Piwik Javascript - cb=" . $cacheBuster . "*/\n";
}
protected function getPreamble()
@ -57,7 +55,7 @@ class JScriptUIAssetMerger extends UIAssetMerger
{
$plugins = $this->getPlugins();
if(!empty($plugins)) {
if (!empty($plugins)) {
/**
* Triggered after all the JavaScript files Piwik uses are minified and merged into a
@ -74,15 +72,16 @@ class JScriptUIAssetMerger extends UIAssetMerger
public function getFileSeparator()
{
return PHP_EOL;
return "\n";
}
protected function processFileContent($uiAsset)
{
$content = $uiAsset->getContent();
if (!$this->assetMinifier->isMinifiedJs($content))
if (!$this->assetMinifier->isMinifiedJs($content)) {
$content = $this->assetMinifier->minifyJs($content);
}
return $content;
}

99
www/analytics/core/AssetManager/UIAssetMerger/StylesheetUIAssetMerger.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,10 +9,10 @@
namespace Piwik\AssetManager\UIAssetMerger;
use Exception;
use lessc;
use Piwik\AssetManager\UIAsset;
use Piwik\AssetManager\UIAssetMerger;
use Piwik\Piwik;
use lessc;
class StylesheetUIAssetMerger extends UIAssetMerger
{
@ -21,7 +21,7 @@ class StylesheetUIAssetMerger extends UIAssetMerger
*/
private $lessCompiler;
function __construct($mergedAsset, $assetFetcher, $cacheBuster)
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
parent::__construct($mergedAsset, $assetFetcher, $cacheBuster);
@ -30,16 +30,10 @@ class StylesheetUIAssetMerger extends UIAssetMerger
protected function getMergedAssets()
{
foreach($this->getAssetCatalog()->getAssets() as $uiAsset) {
$content = $uiAsset->getContent();
if (false !== strpos($content, '@import')) {
$this->lessCompiler->addImportDir(dirname($uiAsset->getAbsoluteLocation()));
}
}
return $this->lessCompiler->compile($this->getConcatenatedAssets());
// note: we're using setImportDir on purpose (not addImportDir)
$this->lessCompiler->setImportDir(PIWIK_USER_PATH);
$concatenatedAssets = $this->getConcatenatedAssets();
return $this->lessCompiler->compile($concatenatedAssets);
}
/**
@ -87,46 +81,71 @@ class StylesheetUIAssetMerger extends UIAssetMerger
protected function processFileContent($uiAsset)
{
return $this->rewriteCssPathsDirectives($uiAsset);
$pathsRewriter = $this->getCssPathsRewriter($uiAsset);
$content = $uiAsset->getContent();
$content = $this->rewriteCssImagePaths($content, $pathsRewriter);
$content = $this->rewriteCssImportPaths($content, $pathsRewriter);
return $content;
}
/**
* Rewrite css url directives
* Rewrite CSS url() directives
*
* @param string $content
* @param callable $pathsRewriter
* @return string
*/
private function rewriteCssImagePaths($content, $pathsRewriter)
{
$content = preg_replace_callback("/(url\(['\"]?)([^'\")]*)/", $pathsRewriter, $content);
return $content;
}
/**
* Rewrite CSS import directives
*
* @param string $content
* @param callable $pathsRewriter
* @return string
*/
private function rewriteCssImportPaths($content, $pathsRewriter)
{
$content = preg_replace_callback("/(@import \")([^\")]*)/", $pathsRewriter, $content);
return $content;
}
/**
* Rewrite CSS url directives
* - rewrites paths defined relatively to their css/less definition file
* - rewrite windows directory separator \\ to /
*
* @param UIAsset $uiAsset
* @return string
* @return \Closure
*/
private function rewriteCssPathsDirectives($uiAsset)
private function getCssPathsRewriter($uiAsset)
{
static $rootDirectoryLength = null;
if (is_null($rootDirectoryLength)) {
$rootDirectoryLength = self::countDirectoriesInPathToRoot($uiAsset);
}
$baseDirectory = dirname($uiAsset->getRelativeLocation());
$content = preg_replace_callback(
"/(url\(['\"]?)([^'\")]*)/",
function ($matches) use ($rootDirectoryLength, $baseDirectory) {
$absolutePath = realpath(PIWIK_USER_PATH . "/$baseDirectory/" . $matches[2]);
return function ($matches) use ($baseDirectory) {
$absolutePath = PIWIK_USER_PATH . "/$baseDirectory/" . $matches[2];
if($absolutePath) {
// Allow to import extension less file
if (strpos($matches[2], '.') === false) {
$absolutePath .= '.less';
}
$relativePath = substr($absolutePath, $rootDirectoryLength);
// Prevent from rewriting full path
$absolutePath = realpath($absolutePath);
if ($absolutePath) {
$relativePath = $baseDirectory . "/" . $matches[2];
$relativePath = str_replace('\\', '/', $relativePath);
$publicPath = $matches[1] . $relativePath;
} else {
$publicPath = $matches[1] . $matches[2];
}
$relativePath = str_replace('\\', '/', $relativePath);
return $matches[1] . $relativePath;
} else {
return $matches[1] . $matches[2];
}
},
$uiAsset->getContent()
);
return $content;
return $publicPath;
};
}
/**
@ -138,7 +157,7 @@ class StylesheetUIAssetMerger extends UIAssetMerger
$rootDirectory = realpath($uiAsset->getBaseDirectory());
if ($rootDirectory != PATH_SEPARATOR
&& substr_compare($rootDirectory, PATH_SEPARATOR, -1)) {
&& substr($rootDirectory, -strlen(PATH_SEPARATOR)) !== PATH_SEPARATOR) {
$rootDirectory .= PATH_SEPARATOR;
}
$rootDirectoryLen = strlen($rootDirectory);

12
www/analytics/core/AssetManager/UIAssetMinifier.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,8 +10,8 @@
namespace Piwik\AssetManager;
use Exception;
use Piwik\Singleton;
use JShrink\Minifier;
use Piwik\Singleton;
class UIAssetMinifier extends Singleton
{
@ -23,7 +23,6 @@ class UIAssetMinifier extends Singleton
parent::__construct();
}
/**
* Indicates if the provided JavaScript content has already been minified or not.
* The heuristic is based on a custom ratio : (size of file) / (number of lines).
@ -37,6 +36,7 @@ class UIAssetMinifier extends Singleton
public function isMinifiedJs($content)
{
$lineCount = substr_count($content, "\n");
if ($lineCount == 0) {
return true;
}
@ -59,8 +59,8 @@ class UIAssetMinifier extends Singleton
private static function validateDependency()
{
if (!class_exists("JShrink\\Minifier"))
throw new Exception("JShrink could not be found, maybe you are using Piwik from git and need to have update Composer. <br>php composer.phar update");
if (!class_exists("JShrink\\Minifier")) {
throw new Exception("JShrink could not be found, maybe you are using Piwik from git and need to update Composer. $ php composer.phar update");
}
}
}

114
www/analytics/core/Auth.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,49 +9,123 @@
namespace Piwik;
use Exception;
/**
* Base for authentication modules
* Base interface for authentication implementations.
*
* Plugins that provide Auth implementations must provide a class that implements
* this interface. Additionally, an instance of that class must be set in the
* container with the 'Piwik\Auth' key during the
* [Request.initAuthenticationObject](http://developer.piwik.org/api-reference/events#requestinitauthenticationobject)
* event.
*
* Authentication implementations must support authentication via username and
* clear-text password and authentication via username and token auth. They can
* additionally support authentication via username and an MD5 hash of a password. If
* they don't support it, then [formless authentication](http://piwik.org/faq/how-to/faq_30/) will fail.
*
* Derived implementations should favor authenticating by password over authenticating
* by token auth. That is to say, if a token auth and a password are set, password
* authentication should be used.
*
* ### Examples
*
* **How an Auth implementation will be used**
*
* // authenticating by password
* $auth = StaticContainer::get('Piwik\Auth');
* $auth->setLogin('user');
* $auth->setPassword('password');
* $result = $auth->authenticate();
*
* // authenticating by token auth
* $auth = StaticContainer::get('Piwik\Auth');
* $auth->setLogin('user');
* $auth->setTokenAuth('...');
* $result = $auth->authenticate();
*
* @api
*/
interface Auth
{
/**
* Authentication module's name, e.g., "Login"
* Must return the Authentication module's name, e.g., `"Login"`.
*
* @return string
*/
public function getName();
/**
* Authenticates user
*
* @return AuthResult
*/
public function authenticate();
/**
* Authenticates the user and initializes the session.
*/
public function initSession($login, $md5Password, $rememberMe);
/**
* Accessor to set authentication token. If set, you can authenticate the tokenAuth by calling the authenticate()
* method afterwards.
* Sets the authentication token to authenticate with.
*
* @param string $token_auth authentication token
*/
public function setTokenAuth($token_auth);
/**
* Accessor to set login name
* Returns the login of the user being authenticated.
*
* @param string $login user login
* @return string
*/
public function getLogin();
/**
* Returns the secret used to calculate a user's token auth.
*
* A users token auth is generated using the user's login and this secret. The secret
* should be specific to the user and not easily guessed. Piwik's default Auth implementation
* uses an MD5 hash of a user's password.
*
* @return string
* @throws Exception if the token auth secret does not exist or cannot be obtained.
*/
public function getTokenAuthSecret();
/**
* Sets the login name to authenticate with.
*
* @param string $login The username.
*/
public function setLogin($login);
/**
* Sets the password to authenticate with.
*
* @param string $password Password (not hashed).
*/
public function setPassword($password);
/**
* Sets the hash of the password to authenticate with. The hash will be an MD5 hash.
*
* @param string $passwordHash The hashed password.
* @throws Exception if authentication by hashed password is not supported.
*/
public function setPasswordHash($passwordHash);
/**
* Authenticates a user using the login and password set using the setters. Can also authenticate
* via token auth if one is set and no password is set.
*
* Note: this method must successfully authenticate if the token auth supplied is a special hash
* of the user's real token auth. This is because the SessionInitializer class stores a
* hash of the token auth in the session cookie. You can calculate the token auth hash using the
* {@link Piwik\Plugins\Login\SessionInitializer::getHashTokenAuth()} method.
*
* @return AuthResult
* @throws Exception if the Auth implementation has an invalid state (ie, no login
* was specified). Note: implementations are not **required** to throw
* exceptions for invalid state, but they are allowed to.
*/
public function authenticate();
}
/**
* Authentication result
* Authentication result. This is what is returned by authentication attempts using {@link Auth}
* implementations.
*
* @api
*/
class AuthResult
{

60
www/analytics/core/BaseFactory.php Normal file
View file

@ -0,0 +1,60 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
/**
* Base class for all factory types.
*
* Factory types are base classes that contain a **factory** method. This method is used to instantiate
* concrete instances by a specified string ID. Fatal errors do not occur if a class does not exist.
* Instead an exception is thrown.
*
* Derived classes should override the **getClassNameFromClassId** and **getInvalidClassIdExceptionMessage**
* static methods.
*/
abstract class BaseFactory
{
/**
* Creates a new instance of a class using a string ID.
*
* @param string $classId The ID of the class.
* @return BaseFactory
* @throws Exception if $classId is invalid.
*/
public static function factory($classId)
{
$className = static::getClassNameFromClassId($classId);
if (!class_exists($className)) {
Common::sendHeader('Content-Type: text/plain; charset=utf-8');
throw new Exception(static::getInvalidClassIdExceptionMessage($classId));
}
return new $className;
}
/**
* Should return a class name based on the class's associated string ID.
*/
protected static function getClassNameFromClassId($id)
{
return $id;
}
/**
* Should return a message to use in an Exception when an invalid class ID is supplied to
* {@link factory()}.
*/
protected static function getInvalidClassIdExceptionMessage($id)
{
return "Invalid class ID '$id' for " . get_called_class() . "::factory().";
}
}

117
www/analytics/core/Cache.php Normal file
View file

@ -0,0 +1,117 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Piwik\Cache\Backend;
use Piwik\Container\StaticContainer;
class Cache
{
/**
* This can be considered as the default cache to use in case you don't know which one to pick. It does not support
* the caching of any objects though. Only boolean, numbers, strings and arrays are supported. Whenever you request
* an entry from the cache it will fetch the entry. Cache entries might be persisted but not necessarily. It
* depends on the configured backend.
*
* @return Cache\Lazy
*/
public static function getLazyCache()
{
return StaticContainer::get('Piwik\Cache\Lazy');
}
/**
* This class is used to cache any data during one request. It won't be persisted between requests and it can
* cache all kind of data, even objects or resources. This cache is very fast.
*
* @return Cache\Transient
*/
public static function getTransientCache()
{
return StaticContainer::get('Piwik\Cache\Transient');
}
/**
* This cache stores all its cache entries under one "cache" entry in a configurable backend.
*
* This comes handy for things that you need very often, nearly in every request. For example plugin metadata, the
* list of tracker plugins, the list of available languages, ...
* Instead of having to read eg. a hundred cache entries from files (or any other backend) it only loads one cache
* entry which contains the hundred keys. Should be used only for things that you need very often and only for
* cache entries that are not too large to keep loading and parsing the single cache entry fast.
* All cache entries it contains have the same life time. For fast performance it won't validate any cache ids.
* It is not possible to cache any objects using this cache.
*
* @return Cache\Eager
*/
public static function getEagerCache()
{
return StaticContainer::get('Piwik\Cache\Eager');
}
public static function flushAll()
{
self::getLazyCache()->flushAll();
self::getTransientCache()->flushAll();
self::getEagerCache()->flushAll();
}
/**
* @param $type
* @return Cache\Backend
*/
public static function buildBackend($type)
{
$factory = new Cache\Backend\Factory();
$options = self::getOptions($type);
$backend = $factory->buildBackend($type, $options);
return $backend;
}
private static function getOptions($type)
{
$options = self::getBackendOptions($type);
switch ($type) {
case 'file':
$options = array('directory' => StaticContainer::get('path.cache'));
break;
case 'chained':
foreach ($options['backends'] as $backend) {
$options[$backend] = self::getOptions($backend);
}
break;
case 'redis':
if (!empty($options['timeout'])) {
$options['timeout'] = (float)Common::forceDotAsSeparatorForDecimalPoint($options['timeout']);
}
break;
}
return $options;
}
private static function getBackendOptions($backend)
{
$key = ucfirst($backend) . 'Cache';
$options = Config::getInstance()->$key;
return $options;
}
}

206
www/analytics/core/CacheFile.php
View file

@ -1,206 +0,0 @@
<?php
/**
* Piwik - Open source web analytics
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
/**
* This class is used to cache data on the filesystem.
*
* It is for example used by the Tracker process to cache various settings and websites attributes in tmp/cache/tracker/*
*
*/
class CacheFile
{
// for testing purposes since tests run on both CLI/FPM (changes in CLI can't invalidate
// opcache in FPM, so we have to invalidate before reading)
public static $invalidateOpCacheBeforeRead = false;
/**
* @var string
*/
protected $cachePath;
/**
* @var
*/
protected $cachePrefix;
/**
* Minimum enforced TTL in seconds
*/
const MINIMUM_TTL = 60;
/**
* @param string $directory directory to use
* @param int $timeToLiveInSeconds TTL
*/
public function __construct($directory, $timeToLiveInSeconds = 300)
{
$cachePath = PIWIK_USER_PATH . '/tmp/cache/' . $directory . '/';
$this->cachePath = SettingsPiwik::rewriteTmpPathWithHostname($cachePath);
if ($timeToLiveInSeconds < self::MINIMUM_TTL) {
$timeToLiveInSeconds = self::MINIMUM_TTL;
}
$this->ttl = $timeToLiveInSeconds;
}
/**
* Function to fetch a cache entry
*
* @param string $id The cache entry ID
* @return array|bool False on error, or array the cache content
*/
public function get($id)
{
if (empty($id)) {
return false;
}
$id = $this->cleanupId($id);
$cache_complete = false;
$content = '';
$expires_on = false;
// We are assuming that most of the time cache will exists
$cacheFilePath = $this->cachePath . $id . '.php';
if (self::$invalidateOpCacheBeforeRead) {
$this->opCacheInvalidate($cacheFilePath);
}
$ok = @include($cacheFilePath);
if ($ok && $cache_complete == true) {
if (empty($expires_on)
|| $expires_on < time()
) {
return false;
}
return $content;
}
return false;
}
private function getExpiresTime()
{
return time() + $this->ttl;
}
protected function cleanupId($id)
{
if (!Filesystem::isValidFilename($id)) {
throw new Exception("Invalid cache ID request $id");
}
return $id;
}
/**
* A function to store content a cache entry.
*
* @param string $id The cache entry ID
* @param array $content The cache content
* @throws \Exception
* @return bool True if the entry was succesfully stored
*/
public function set($id, $content)
{
if (empty($id)) {
return false;
}
if (!is_dir($this->cachePath)) {
Filesystem::mkdir($this->cachePath);
}
if (!is_writable($this->cachePath)) {
return false;
}
$id = $this->cleanupId($id);
$id = $this->cachePath . $id . '.php';
if (is_object($content)) {
throw new \Exception('You cannot use the CacheFile to cache an object, only arrays, strings and numbers.');
}
$cache_literal = "<" . "?php\n";
$cache_literal .= "$" . "content = " . var_export($content, true) . ";\n";
$cache_literal .= "$" . "expires_on = " . $this->getExpiresTime() . ";\n";
$cache_literal .= "$" . "cache_complete = true;\n";
$cache_literal .= "?" . ">";
// Write cache to a temp file, then rename it, overwriting the old cache
// On *nix systems this should guarantee atomicity
$tmp_filename = tempnam($this->cachePath, 'tmp_');
@chmod($tmp_filename, 0640);
if ($fp = @fopen($tmp_filename, 'wb')) {
@fwrite($fp, $cache_literal, strlen($cache_literal));
@fclose($fp);
if (!@rename($tmp_filename, $id)) {
// On some systems rename() doesn't overwrite destination
@unlink($id);
if (!@rename($tmp_filename, $id)) {
// Make sure that no temporary file is left over
// if the destination is not writable
@unlink($tmp_filename);
}
}
$this->opCacheInvalidate($id);
return true;
}
return false;
}
/**
* A function to delete a single cache entry
*
* @param string $id The cache entry ID
* @return bool True if the entry was succesfully deleted
*/
public function delete($id)
{
if (empty($id)) {
return false;
}
$id = $this->cleanupId($id);
$filename = $this->cachePath . $id . '.php';
if (file_exists($filename)) {
$this->opCacheInvalidate($filename);
@unlink($filename);
return true;
}
return false;
}
/**
* A function to delete all cache entries in the directory
*/
public function deleteAll()
{
$self = $this;
$beforeUnlink = function ($path) use ($self) {
$self->opCacheInvalidate($path);
};
Filesystem::unlinkRecursive($this->cachePath, $deleteRootToo = false, $beforeUnlink);
}
public function opCacheInvalidate($filepath)
{
if (function_exists('opcache_invalidate')
&& is_file($filepath)
) {
opcache_invalidate($filepath, $force = true);
}
}
}

29
www/analytics/core/CacheId.php Normal file
View file

@ -0,0 +1,29 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Piwik\Plugin\Manager;
class CacheId
{
public static function languageAware($cacheId)
{
return $cacheId . '-' . Translate::getLanguageLoaded();
}
public static function pluginAware($cacheId)
{
$pluginManager = Manager::getInstance();
$pluginNames = $pluginManager->getLoadedPluginsName();
$cacheId = $cacheId . '-' . md5(implode('', $pluginNames));
$cacheId = self::languageAware($cacheId);
return $cacheId;
}
}

247
www/analytics/core/CliMulti.php
View file

@ -1,19 +1,23 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik;
use Piwik\Archiver\Request;
use Piwik\CliMulti\CliPhp;
use Piwik\CliMulti\Output;
use Piwik\CliMulti\Process;
use Piwik\Container\StaticContainer;
/**
* Class CliMulti.
*/
class CliMulti {
class CliMulti
{
/**
* If set to true or false it will overwrite whether async is supported or not.
@ -23,17 +27,37 @@ class CliMulti {
public $supportsAsync = null;
/**
* @var \Piwik\CliMulti\Process[]
* @var Process[]
*/
private $processes = array();
/**
* @var \Piwik\CliMulti\Output[]
* If set it will issue at most concurrentProcessesLimit requests
* @var int
*/
private $concurrentProcessesLimit = null;
/**
* @var Output[]
*/
private $outputs = array();
private $acceptInvalidSSLCertificate = false;
/**
* @var bool
*/
private $runAsSuperUser = false;
/**
* Only used when doing synchronous curl requests.
*
* @var string
*/
private $urlToPiwik = null;
private $phpCliOptions = '';
public function __construct()
{
$this->supportsAsync = $this->supportsAsync();
@ -44,26 +68,38 @@ class CliMulti {
* If multi cli is not supported (eg windows) it will initiate an HTTP request instead (not async).
*
* @param string[] $piwikUrls An array of urls, for instance:
* array('http://www.example.com/piwik?module=API...')
*
* `array('http://www.example.com/piwik?module=API...')`
*
* **Make sure query parameter values are properly encoded in the URLs.**
*
* @return array The response of each URL in the same order as the URLs. The array can contain null values in case
* there was a problem with a request, for instance if the process died unexpected.
*/
public function request(array $piwikUrls)
{
$this->start($piwikUrls);
$chunks = array($piwikUrls);
if ($this->concurrentProcessesLimit) {
$chunks = array_chunk($piwikUrls, $this->concurrentProcessesLimit);
}
do {
usleep(100000); // 100 * 1000 = 100ms
} while (!$this->hasFinished());
$results = $this->getResponse($piwikUrls);
$this->cleanup();
self::cleanupNotRemovedFiles();
$results = array();
foreach ($chunks as $urlsChunk) {
$results = array_merge($results, $this->requestUrls($urlsChunk));
}
return $results;
}
/**
* Forwards the given configuration options to the PHP cli command.
* @param string $phpCliOptions eg "-d memory_limit=8G -c=path/to/php.ini"
*/
public function setPhpCliConfigurationOptions($phpCliOptions)
{
$this->phpCliOptions = (string) $phpCliOptions;
}
/**
* Ok, this sounds weird. Why should we care about ssl certificates when we are in CLI mode? It is needed for
* our simple fallback mode for Windows where we initiate HTTP requests instead of CLI.
@ -74,28 +110,51 @@ class CliMulti {
$this->acceptInvalidSSLCertificate = $acceptInvalidSSLCertificate;
}
/**
* @param $limit int Maximum count of requests to issue in parallel
*/
public function setConcurrentProcessesLimit($limit)
{
$this->concurrentProcessesLimit = $limit;
}
public function runAsSuperUser($runAsSuperUser = true)
{
$this->runAsSuperUser = $runAsSuperUser;
}
private function start($piwikUrls)
{
foreach ($piwikUrls as $index => $url) {
$cmdId = $this->generateCommandId($url) . $index;
$output = new Output($cmdId);
if ($this->supportsAsync) {
$this->executeAsyncCli($url, $output, $cmdId);
} else {
$this->executeNotAsyncHttp($url, $output);
if ($url instanceof Request) {
$url->start();
}
$this->outputs[] = $output;
$cmdId = $this->generateCommandId($url) . $index;
$this->executeUrlCommand($cmdId, $url);
}
}
private function executeUrlCommand($cmdId, $url)
{
$output = new Output($cmdId);
if ($this->supportsAsync) {
$this->executeAsyncCli($url, $output, $cmdId);
} else {
$this->executeNotAsyncHttp($url, $output);
}
$this->outputs[] = $output;
}
private function buildCommand($hostname, $query, $outputFile)
{
$bin = $this->findPhpBinary();
$superuserCommand = $this->runAsSuperUser ? "--superuser" : "";
return sprintf('%s -q %s/console climulti:request --piwik-domain=%s %s > %s 2>&1 &',
$bin, PIWIK_INCLUDE_PATH, escapeshellarg($hostname), escapeshellarg($query), $outputFile);
return sprintf('%s %s %s/console climulti:request -q --piwik-domain=%s %s %s > %s 2>&1 &',
$bin, $this->phpCliOptions, PIWIK_INCLUDE_PATH, escapeshellarg($hostname), $superuserCommand, escapeshellarg($query), $outputFile);
}
private function getResponse()
@ -119,7 +178,6 @@ class CliMulti {
// ==> declare the process as finished
$process->finishProcess();
continue;
} elseif (!$hasStarted) {
return false;
}
@ -128,6 +186,14 @@ class CliMulti {
return false;
}
$pid = $process->getPid();
foreach ($this->outputs as $output) {
if ($output->getOutputId() === $pid && $output->isAbnormal()) {
$process->finishProcess();
return true;
}
}
if ($process->hasFinished()) {
// prevent from checking this process over and over again
unset($this->processes[$index]);
@ -146,11 +212,15 @@ class CliMulti {
* What is missing under windows? Detection whether a process is still running in Process::isProcessStillRunning
* and how to send a process into background in start()
*/
private function supportsAsync()
public function supportsAsync()
{
return !SettingsServer::isWindows()
&& Process::isSupported()
&& $this->findPhpBinary();
return Process::isSupported() && !Common::isPhpCgiType() && $this->findPhpBinary();
}
private function findPhpBinary()
{
$cliPhp = new CliPhp();
return $cliPhp->findPhpBinary();
}
private function cleanup()
@ -176,61 +246,33 @@ class CliMulti {
$timeOneWeekAgo = strtotime('-1 week');
$files = _glob(self::getTmpPath() . '/*');
if(empty($files)) {
if (empty($files)) {
return;
}
foreach ($files as $file) {
$timeLastModified = filemtime($file);
if (file_exists($file)) {
$timeLastModified = filemtime($file);
if ($timeOneWeekAgo > $timeLastModified) {
unlink($file);
if ($timeLastModified !== false && $timeOneWeekAgo > $timeLastModified) {
unlink($file);
}
}
}
}
public static function getTmpPath()
{
$dir = PIWIK_INCLUDE_PATH . '/tmp/climulti';
return SettingsPiwik::rewriteTmpPathWithHostname($dir);
}
private function findPhpBinary()
{
if (defined('PHP_BINARY') && false === strpos(PHP_BINARY, 'fpm')) {
return PHP_BINARY;
}
$bin = '';
if (!empty($_SERVER['_']) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['_']);
}
if (empty($bin) && !empty($_SERVER['argv'][0]) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['argv'][0]);
}
if (empty($bin)) {
$bin = shell_exec('which php');
}
if (empty($bin)) {
$bin = shell_exec('which php5');
}
if (!empty($bin)) {
return trim($bin);
}
return StaticContainer::get('path.tmp') . '/climulti';
}
private function executeAsyncCli($url, Output $output, $cmdId)
{
$this->processes[] = new Process($cmdId);
$url = $this->appendTestmodeParamToUrlIfNeeded($url);
$query = UrlHelper::getQueryFromUrl($url, array('pid' => $cmdId));
$hostname = UrlHelper::getHostFromUrl($url);
$url = $this->appendTestmodeParamToUrlIfNeeded($url);
$query = UrlHelper::getQueryFromUrl($url, array('pid' => $cmdId));
$hostname = Url::getHost($checkIfTrusted = false);
$command = $this->buildCommand($hostname, $query, $output->getPathToFile());
Log::debug($command);
@ -239,6 +281,29 @@ class CliMulti {
private function executeNotAsyncHttp($url, Output $output)
{
$piwikUrl = $this->urlToPiwik ?: SettingsPiwik::getPiwikUrl();
if (empty($piwikUrl)) {
$piwikUrl = 'http://' . Url::getHost() . '/';
}
$url = $piwikUrl . $url;
if (Config::getInstance()->General['force_ssl'] == 1) {
$url = str_replace("http://", "https://", $url);
}
if ($this->runAsSuperUser) {
$tokenAuths = self::getSuperUserTokenAuths();
$tokenAuth = reset($tokenAuths);
if (strpos($url, '?') === false) {
$url .= '?';
} else {
$url .= '&';
}
$url .= 'token_auth=' . $tokenAuth;
}
try {
Log::debug("Execute HTTP API request: " . $url);
$response = Http::sendHttpRequestBy('curl', $url, $timeout = 0, $userAgent = null, $destinationPath = null, $file = null, $followDepth = 0, $acceptLanguage = false, $this->acceptInvalidSSLCertificate);
@ -246,19 +311,21 @@ class CliMulti {
} catch (\Exception $e) {
$message = "Got invalid response from API request: $url. ";
if (empty($response)) {
if (isset($response) && empty($response)) {
$message .= "The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.";
} else {
$message .= "Response was '" . $e->getMessage() . "'";
}
$output->write($message);
Log::debug($e);
}
}
private function appendTestmodeParamToUrlIfNeeded($url)
{
$isTestMode = $url && false !== strpos($url, 'tests/PHPUnit/proxy');
$isTestMode = defined('PIWIK_TEST_MODE');
if ($isTestMode && false === strpos($url, '?')) {
$url .= "?testmode=1";
@ -269,12 +336,42 @@ class CliMulti {
return $url;
}
private function getPhpCommandIfValid($path)
/**
* @param array $piwikUrls
* @return array
*/
private function requestUrls(array $piwikUrls)
{
if (!empty($path) && is_executable($path)) {
if (0 === strpos($path, PHP_BINDIR) && false === strpos($path, 'phpunit')) {
return $path;
}
}
$this->start($piwikUrls);
do {
usleep(100000); // 100 * 1000 = 100ms
} while (!$this->hasFinished());
$results = $this->getResponse($piwikUrls);
$this->cleanup();
self::cleanupNotRemovedFiles();
return $results;
}
private static function getSuperUserTokenAuths()
{
$tokens = array();
/**
* Used to be in CronArchive, moved to CliMulti.
*
* @ignore
*/
Piwik::postEvent('CronArchive.getTokenAuth', array(&$tokens));
return $tokens;
}
public function setUrlToPiwik($urlToPiwik)
{
$this->urlToPiwik = $urlToPiwik;
}
}

101
www/analytics/core/CliMulti/CliPhp.php Normal file
View file

@ -0,0 +1,101 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\Common;
class CliPhp
{
public function findPhpBinary()
{
if (defined('PHP_BINARY')) {
if ($this->isHhvmBinary(PHP_BINARY)) {
return PHP_BINARY . ' --php';
}
if ($this->isValidPhpType(PHP_BINARY)) {
return PHP_BINARY . ' -q';
}
}
$bin = '';
if (!empty($_SERVER['_']) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['_']);
}
if (empty($bin) && !empty($_SERVER['argv'][0]) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['argv'][0]);
}
if (!$this->isValidPhpType($bin)) {
$bin = shell_exec('which php');
}
if (!$this->isValidPhpType($bin)) {
$bin = shell_exec('which php5');
}
if (!$this->isValidPhpType($bin)) {
return false;
}
$bin = trim($bin);
if (!$this->isValidPhpVersion($bin)) {
return false;
}
$bin .= ' -q';
return $bin;
}
private function isHhvmBinary($bin)
{
return false !== strpos($bin, 'hhvm');
}
private function isValidPhpVersion($bin)
{
global $piwik_minimumPHPVersion;
$cliVersion = $this->getPhpVersion($bin);
$isCliVersionValid = version_compare($piwik_minimumPHPVersion, $cliVersion) <= 0;
return $isCliVersionValid;
}
private function isValidPhpType($path)
{
return !empty($path)
&& false === strpos($path, 'fpm')
&& false === strpos($path, 'cgi')
&& false === strpos($path, 'phpunit');
}
private function getPhpCommandIfValid($path)
{
if (!empty($path) && is_executable($path)) {
if (0 === strpos($path, PHP_BINDIR) && $this->isValidPhpType($path)) {
return $path;
}
}
return null;
}
/**
* @param string $bin PHP binary
* @return string
*/
private function getPhpVersion($bin)
{
$command = sprintf("%s -r 'echo phpversion();'", $bin);
$version = shell_exec($command);
return $version;
}
}

25
www/analytics/core/CliMulti/Output.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,9 +10,11 @@ namespace Piwik\CliMulti;
use Piwik\CliMulti;
use Piwik\Filesystem;
class Output {
class Output
{
private $tmpFile = '';
private $outputId = null;
public function __construct($outputId)
{
@ -21,8 +23,15 @@ class Output {
}
$dir = CliMulti::getTmpPath();
Filesystem::mkdir($dir, true);
$this->tmpFile = $dir . '/' . $outputId . '.output';
Filesystem::mkdir($dir);
$this->tmpFile = $dir . '/' . $outputId . '.output';
$this->outputId = $outputId;
}
public function getOutputId()
{
return $this->outputId;
}
public function write($content)
@ -35,6 +44,13 @@ class Output {
return $this->tmpFile;
}
public function isAbnormal()
{
$size = Filesystem::getFileSize($this->tmpFile, 'MB');
return $size !== null && $size >= 100;
}
public function exists()
{
return file_exists($this->tmpFile);
@ -49,5 +65,4 @@ class Output {
{
Filesystem::deleteFileIfExists($this->tmpFile);
}
}

84
www/analytics/core/CliMulti/Process.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -24,6 +24,7 @@ class Process
private $pidFile = '';
private $timeCreation = null;
private $isSupported = null;
private $pid = null;
public function __construct($pid)
{
@ -32,15 +33,21 @@ class Process
}
$pidDir = CliMulti::getTmpPath();
Filesystem::mkdir($pidDir, true);
Filesystem::mkdir($pidDir);
$this->isSupported = self::isSupported();
$this->pidFile = $pidDir . '/' . $pid . '.pid';
$this->timeCreation = time();
$this->pid = $pid;
$this->markAsNotStarted();
}
public function getPid()
{
return $this->pid;
}
private function markAsNotStarted()
{
$content = $this->getPidFileContent();
@ -97,6 +104,11 @@ class Process
return false;
}
if (!$this->pidFileSizeIsNormal()) {
$this->finishProcess();
return false;
}
if ($this->isProcessStillRunning($content)) {
return true;
}
@ -108,6 +120,13 @@ class Process
return false;
}
private function pidFileSizeIsNormal()
{
$size = Filesystem::getFileSize($this->pidFile);
return $size !== null && $size < 500;
}
public function finishProcess()
{
Filesystem::deleteFileIfExists($this->pidFile);
@ -125,7 +144,7 @@ class Process
}
$lockedPID = trim($content);
$runningPIDs = explode("\n", trim( `ps -e | awk '{print $1}'` ));
$runningPIDs = self::getRunningProcesses();
return !empty($lockedPID) && in_array($lockedPID, $runningPIDs);
}
@ -154,17 +173,30 @@ class Process
return false;
}
if (static::commandExists('ps') && self::returnsSuccessCode('ps') && self::commandExists('awk')) {
if (!self::commandExists('ps') || !self::returnsSuccessCode('ps') || !self::commandExists('awk')) {
return false;
}
if (count(self::getRunningProcesses()) > 0) {
return true;
}
return false;
if (!self::isProcFSMounted()) {
return false;
}
return true;
}
private static function isSystemNotSupported()
{
$uname = shell_exec('uname -a');
if(strpos($uname, 'synology') !== false) {
$uname = @shell_exec('uname -a 2> /dev/null');
if (empty($uname)) {
$uname = php_uname();
}
if (strpos($uname, 'synology') !== false) {
return true;
}
return false;
@ -175,12 +207,12 @@ class Process
$command = 'shell_exec';
$disabled = explode(',', ini_get('disable_functions'));
$disabled = array_map('trim', $disabled);
return in_array($command, $disabled);
return in_array($command, $disabled) || !function_exists($command);
}
private static function returnsSuccessCode($command)
{
$exec = $command . ' > /dev/null 2>&1 & echo $?';
$exec = $command . ' > /dev/null 2>&1; echo $?';
$returnCode = shell_exec($exec);
$returnCode = trim($returnCode);
return 0 == (int) $returnCode;
@ -188,8 +220,38 @@ class Process
private static function commandExists($command)
{
$result = shell_exec('which ' . escapeshellarg($command));
$result = @shell_exec('which ' . escapeshellarg($command) . ' 2> /dev/null');
return !empty($result);
}
}
/**
* ps -e requires /proc
* @return bool
*/
private static function isProcFSMounted()
{
if (is_resource(@fopen('/proc', 'r'))) {
return true;
}
// Testing if /proc is a resource with @fopen fails on systems with open_basedir set.
// by using stat we not only test the existance of /proc but also confirm it's a 'proc' filesystem
$type = @shell_exec('stat -f -c "%T" /proc 2>/dev/null');
return strpos($type, 'proc') === 0;
}
/**
* @return int[] The ids of the currently running processes
*/
public static function getRunningProcesses()
{
$ids = explode("\n", trim(`ps ex 2>/dev/null | awk '{print $1}' 2>/dev/null`));
$ids = array_map('intval', $ids);
$ids = array_filter($ids, function ($id) {
return $id > 0;
});
return $ids;
}
}

64
www/analytics/core/CliMulti/RequestCommand.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,6 +8,12 @@
namespace Piwik\CliMulti;
use Piwik\Application\Environment;
use Piwik\Access;
use Piwik\Container\StaticContainer;
use Piwik\Db;
use Piwik\Log;
use Piwik\Option;
use Piwik\Plugin\ConsoleCommand;
use Piwik\Url;
use Piwik\UrlHelper;
@ -15,33 +21,41 @@ use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Piwik\Config;
use Piwik\Common;
use Piwik\FrontController;
/**
* RequestCommand
*/
class RequestCommand extends ConsoleCommand
{
/**
* @var Environment
*/
private $environment;
protected function configure()
{
$this->setName('climulti:request');
$this->setDescription('Parses and executes the given query. See Piwik\CliMulti. Intended only for system usage.');
$this->addArgument('url-query', null, InputOption::VALUE_REQUIRED, 'Piwik URL query string, for instance: "module=API&method=API.getPiwikVersion&token_auth=123456789"');
$this->addArgument('url-query', InputArgument::REQUIRED, 'Piwik URL query string, for instance: "module=API&method=API.getPiwikVersion&token_auth=123456789"');
$this->addOption('superuser', null, InputOption::VALUE_NONE, 'If supplied, runs the code as superuser.');
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$this->recreateContainerWithWebEnvironment();
$this->initHostAndQueryString($input);
if ($this->isTestModeEnabled()) {
Config::getInstance()->setTestEnvironment();
$indexFile = '/tests/PHPUnit/proxy/index.php';
$indexFile = '/tests/PHPUnit/proxy/';
$this->resetDatabase();
} else {
$indexFile = '/index.php';
$indexFile = '/';
}
$indexFile .= 'index.php';
if (!empty($_GET['pid'])) {
$process = new Process($_GET['pid']);
@ -52,6 +66,16 @@ class RequestCommand extends ConsoleCommand
$process->startProcess();
}
if ($input->getOption('superuser')) {
StaticContainer::addDefinitions(array(
'observers.global' => \DI\add(array(
array('Environment.bootstrapped', function () {
Access::getInstance()->setSuperUserAccess(true);
})
)),
));
}
require_once PIWIK_INCLUDE_PATH . $indexFile;
if (!empty($process)) {
@ -75,10 +99,30 @@ class RequestCommand extends ConsoleCommand
Url::setHost($hostname);
$query = $input->getArgument('url-query');
$query = UrlHelper::getArrayFromQueryString($query);
$query = UrlHelper::getArrayFromQueryString($query); // NOTE: this method can create the StaticContainer now
foreach ($query as $name => $value) {
$_GET[$name] = $value;
}
}
}
/**
* We will be simulating an HTTP request here (by including index.php).
*
* To avoid weird side-effects (e.g. the logging output messing up the HTTP response on the CLI output)
* we need to recreate the container with the default environment instead of the CLI environment.
*/
private function recreateContainerWithWebEnvironment()
{
StaticContainer::clearContainer();
Log::unsetInstance();
$this->environment = new Environment(null);
$this->environment->init();
}
private function resetDatabase()
{
Option::clearCache();
Db::destroyDatabaseObject();
}
}

242
www/analytics/core/Columns/Dimension.php Normal file
View file

@ -0,0 +1,242 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Columns;
use Exception;
use Piwik\Plugin;
use Piwik\Plugin\ComponentFactory;
use Piwik\Plugin\Dimension\ActionDimension;
use Piwik\Plugin\Dimension\ConversionDimension;
use Piwik\Plugin\Dimension\VisitDimension;
use Piwik\Plugin\Segment;
/**
* @api
* @since 2.5.0
*/
abstract class Dimension
{
const COMPONENT_SUBNAMESPACE = 'Columns';
// TODO that we have quite a few @ignore in public methods might show we should maybe split some code into two
// classes.
/**
* This will be the name of the column in the database table if a $columnType is specified.
* @var string
* @api
*/
protected $columnName = '';
/**
* If a columnType is defined, we will create a column in the MySQL table having this type. Please make sure
* MySQL understands this type. Once you change the column type the Piwik platform will notify the user to
* perform an update which can sometimes take a long time so be careful when choosing the correct column type.
* @var string
* @api
*/
protected $columnType = '';
/**
* Holds an array of segment instances
* @var Segment[]
*/
protected $segments = array();
/**
* Overwrite this method to configure segments. To do so just create an instance of a {@link \Piwik\Plugin\Segment}
* class, configure it and call the {@link addSegment()} method. You can add one or more segments for this
* dimension. Example:
*
* ```
* $segment = new Segment();
* $segment->setSegment('exitPageUrl');
* $segment->setName('Actions_ColumnExitPageURL');
* $segment->setCategory('General_Visit');
* $this->addSegment($segment);
* ```
*/
protected function configureSegments()
{
}
/**
* Check whether a dimension has overwritten a specific method.
* @param $method
* @return bool
* @ignore
*/
public function hasImplementedEvent($method)
{
$method = new \ReflectionMethod($this, $method);
$declaringClass = $method->getDeclaringClass();
return 0 === strpos($declaringClass->name, 'Piwik\Plugins');
}
/**
* Adds a new segment. The segment type will be set to 'dimension' automatically if not already set.
* @param Segment $segment
* @api
*/
protected function addSegment(Segment $segment)
{
$type = $segment->getType();
if (empty($type)) {
$segment->setType(Segment::TYPE_DIMENSION);
}
$this->segments[] = $segment;
}
/**
* Get the list of configured segments.
* @return Segment[]
* @ignore
*/
public function getSegments()
{
if (empty($this->segments)) {
$this->configureSegments();
}
return $this->segments;
}
/**
* Get the name of the dimension column.
* @return string
* @ignore
*/
public function getColumnName()
{
return $this->columnName;
}
/**
* Check whether the dimension has a column type configured
* @return bool
* @ignore
*/
public function hasColumnType()
{
return !empty($this->columnType);
}
/**
* Get the translated name of the dimension. Defaults to an empty string.
* @return string
* @api
*/
public function getName()
{
return '';
}
/**
* Returns a unique string ID for this dimension. The ID is built using the namespaced class name
* of the dimension, but is modified to be more human readable.
*
* @return string eg, `"Referrers.Keywords"`
* @throws Exception if the plugin and simple class name of this instance cannot be determined.
* This would only happen if the dimension is located in the wrong directory.
* @api
*/
public function getId()
{
$className = get_class($this);
// parse plugin name & dimension name
$regex = "/Piwik\\\\Plugins\\\\([^\\\\]+)\\\\" . self::COMPONENT_SUBNAMESPACE . "\\\\([^\\\\]+)/";
if (!preg_match($regex, $className, $matches)) {
throw new Exception("'$className' is located in the wrong directory.");
}
$pluginName = $matches[1];
$dimensionName = $matches[2];
return $pluginName . '.' . $dimensionName;
}
/**
* Gets an instance of all available visit, action and conversion dimension.
* @return Dimension[]
*/
public static function getAllDimensions()
{
$dimensions = array();
foreach (VisitDimension::getAllDimensions() as $dimension) {
$dimensions[] = $dimension;
}
foreach (ActionDimension::getAllDimensions() as $dimension) {
$dimensions[] = $dimension;
}
foreach (ConversionDimension::getAllDimensions() as $dimension) {
$dimensions[] = $dimension;
}
return $dimensions;
}
public static function getDimensions(Plugin $plugin)
{
$dimensions = array();
foreach (VisitDimension::getDimensions($plugin) as $dimension) {
$dimensions[] = $dimension;
}
foreach (ActionDimension::getDimensions($plugin) as $dimension) {
$dimensions[] = $dimension;
}
foreach (ConversionDimension::getDimensions($plugin) as $dimension) {
$dimensions[] = $dimension;
}
return $dimensions;
}
/**
* Creates a Dimension instance from a string ID (see {@link getId()}).
*
* @param string $dimensionId See {@link getId()}.
* @return Dimension|null The created instance or null if there is no Dimension for
* $dimensionId or if the plugin that contains the Dimension is
* not loaded.
* @api
*/
public static function factory($dimensionId)
{
list($module, $dimension) = explode('.', $dimensionId);
return ComponentFactory::factory($module, $dimension, __CLASS__);
}
/**
* Returns the name of the plugin that contains this Dimension.
*
* @return string
* @throws Exception if the Dimension is not located within a Plugin module.
* @api
*/
public function getModule()
{
$id = $this->getId();
if (empty($id)) {
throw new Exception("Invalid dimension ID: '$id'.");
}
$parts = explode('.', $id);
return reset($parts);
}
}

372
www/analytics/core/Columns/Updater.php Normal file
View file

@ -0,0 +1,372 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Columns;
use Piwik\Common;
use Piwik\DbHelper;
use Piwik\Plugin\Dimension\ActionDimension;
use Piwik\Plugin\Dimension\VisitDimension;
use Piwik\Plugin\Dimension\ConversionDimension;
use Piwik\Db;
use Piwik\Updater as PiwikUpdater;
use Piwik\Filesystem;
use Piwik\Cache as PiwikCache;
/**
* Class that handles dimension updates
*/
class Updater extends \Piwik\Updates
{
private static $cacheId = 'AllDimensionModifyTime';
/**
* @var VisitDimension[]
*/
public $visitDimensions;
/**
* @var ActionDimension[]
*/
private $actionDimensions;
/**
* @var ConversionDimension[]
*/
private $conversionDimensions;
/**
* @param VisitDimension[]|null $visitDimensions
* @param ActionDimension[]|null $actionDimensions
* @param ConversionDimension[]|null $conversionDimensions
*/
public function __construct(array $visitDimensions = null, array $actionDimensions = null, array $conversionDimensions = null)
{
$this->visitDimensions = $visitDimensions;
$this->actionDimensions = $actionDimensions;
$this->conversionDimensions = $conversionDimensions;
}
public function getMigrationQueries(PiwikUpdater $updater)
{
$sqls = array();
$changingColumns = $this->getUpdates($updater);
foreach ($changingColumns as $table => $columns) {
if (empty($columns) || !is_array($columns)) {
continue;
}
$sqls["ALTER TABLE `" . Common::prefixTable($table) . "` " . implode(', ', $columns)] = array('1091', '1060');
}
return $sqls;
}
public function doUpdate(PiwikUpdater $updater)
{
$updater->executeMigrationQueries(__FILE__, $this->getMigrationQueries($updater));
}
private function getVisitDimensions()
{
// see eg https://github.com/piwik/piwik/issues/8399 we fetch them only on demand to improve performance
if (!isset($this->visitDimensions)) {
$this->visitDimensions = VisitDimension::getAllDimensions();
}
return $this->visitDimensions;
}
private function getActionDimensions()
{
// see eg https://github.com/piwik/piwik/issues/8399 we fetch them only on demand to improve performance
if (!isset($this->actionDimensions)) {
$this->actionDimensions = ActionDimension::getAllDimensions();
}
return $this->actionDimensions;
}
private function getConversionDimensions()
{
// see eg https://github.com/piwik/piwik/issues/8399 we fetch them only on demand to improve performance
if (!isset($this->conversionDimensions)) {
$this->conversionDimensions = ConversionDimension::getAllDimensions();
}
return $this->conversionDimensions;
}
private function getUpdates(PiwikUpdater $updater)
{
$visitColumns = DbHelper::getTableColumns(Common::prefixTable('log_visit'));
$actionColumns = DbHelper::getTableColumns(Common::prefixTable('log_link_visit_action'));
$conversionColumns = DbHelper::getTableColumns(Common::prefixTable('log_conversion'));
$allUpdatesToRun = array();
foreach ($this->getVisitDimensions() as $dimension) {
$updates = $this->getUpdatesForDimension($updater, $dimension, 'log_visit.', $visitColumns, $conversionColumns);
$allUpdatesToRun = $this->mixinUpdates($allUpdatesToRun, $updates);
}
foreach ($this->getActionDimensions() as $dimension) {
$updates = $this->getUpdatesForDimension($updater, $dimension, 'log_link_visit_action.', $actionColumns);
$allUpdatesToRun = $this->mixinUpdates($allUpdatesToRun, $updates);
}
foreach ($this->getConversionDimensions() as $dimension) {
$updates = $this->getUpdatesForDimension($updater, $dimension, 'log_conversion.', $conversionColumns);
$allUpdatesToRun = $this->mixinUpdates($allUpdatesToRun, $updates);
}
return $allUpdatesToRun;
}
/**
* @param ActionDimension|ConversionDimension|VisitDimension $dimension
* @param string $componentPrefix
* @param array $existingColumnsInDb
* @param array $conversionColumns
* @return array
*/
private function getUpdatesForDimension(PiwikUpdater $updater, $dimension, $componentPrefix, $existingColumnsInDb, $conversionColumns = array())
{
$column = $dimension->getColumnName();
$componentName = $componentPrefix . $column;
if (!$updater->hasNewVersion($componentName)) {
return array();
}
if (array_key_exists($column, $existingColumnsInDb)) {
if ($dimension instanceof VisitDimension) {
$sqlUpdates = $dimension->update($conversionColumns);
} else {
$sqlUpdates = $dimension->update();
}
} else {
$sqlUpdates = $dimension->install();
}
return $sqlUpdates;
}
private function mixinUpdates($allUpdatesToRun, $updatesFromDimension)
{
if (!empty($updatesFromDimension)) {
foreach ($updatesFromDimension as $table => $col) {
if (empty($allUpdatesToRun[$table])) {
$allUpdatesToRun[$table] = $col;
} else {
$allUpdatesToRun[$table] = array_merge($allUpdatesToRun[$table], $col);
}
}
}
return $allUpdatesToRun;
}
public function getAllVersions(PiwikUpdater $updater)
{
// to avoid having to load all dimensions on each request we check if there were any changes on the file system
// can easily save > 100ms for each request
$cachedTimes = self::getCachedDimensionFileChanges();
$currentTimes = self::getCurrentDimensionFileChanges();
$diff = array_diff_assoc($currentTimes, $cachedTimes);
if (empty($diff)) {
return array();
}
$versions = array();
$visitColumns = DbHelper::getTableColumns(Common::prefixTable('log_visit'));
$actionColumns = DbHelper::getTableColumns(Common::prefixTable('log_link_visit_action'));
$conversionColumns = DbHelper::getTableColumns(Common::prefixTable('log_conversion'));
foreach ($this->getVisitDimensions() as $dimension) {
$versions = $this->mixinVersions($updater, $dimension, VisitDimension::INSTALLER_PREFIX, $visitColumns, $versions);
}
foreach ($this->getActionDimensions() as $dimension) {
$versions = $this->mixinVersions($updater, $dimension, ActionDimension::INSTALLER_PREFIX, $actionColumns, $versions);
}
foreach ($this->getConversionDimensions() as $dimension) {
$versions = $this->mixinVersions($updater, $dimension, ConversionDimension::INSTALLER_PREFIX, $conversionColumns, $versions);
}
return $versions;
}
/**
* @param ActionDimension|ConversionDimension|VisitDimension $dimension
* @param string $componentPrefix
* @param array $columns
* @param array $versions
* @return array The modified versions array
*/
private function mixinVersions(PiwikUpdater $updater, $dimension, $componentPrefix, $columns, $versions)
{
$columnName = $dimension->getColumnName();
// dimensions w/o columns do not need DB updates
if (!$columnName || !$dimension->hasColumnType()) {
return $versions;
}
$component = $componentPrefix . $columnName;
$version = $dimension->getVersion();
// if the column exists in the table, but has no associated version, and was one of the core columns
// that was moved when the dimension refactor took place, then:
// - set the installed version in the DB to the current code version
// - and do not check for updates since we just set the version to the latest
if (array_key_exists($columnName, $columns)
&& false === $updater->getCurrentComponentVersion($component)
&& self::wasDimensionMovedFromCoreToPlugin($component, $version)
) {
$updater->markComponentSuccessfullyUpdated($component, $version);
return $versions;
}
$versions[$component] = $version;
return $versions;
}
public static function isDimensionComponent($name)
{
return 0 === strpos($name, 'log_visit.')
|| 0 === strpos($name, 'log_conversion.')
|| 0 === strpos($name, 'log_conversion_item.')
|| 0 === strpos($name, 'log_link_visit_action.');
}
public static function wasDimensionMovedFromCoreToPlugin($name, $version)
{
// maps names of core dimension columns that were part of the original dimension refactor with their
// initial "version" strings. The '1' that is sometimes appended to the end of the string (sometimes seen as
// NULL1) is from individual dimension "versioning" logic (eg, see VisitDimension::getVersion())
$initialCoreDimensionVersions = array(
'log_visit.config_resolution' => 'VARCHAR(9) NOT NULL',
'log_visit.config_device_brand' => 'VARCHAR( 100 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL',
'log_visit.config_device_model' => 'VARCHAR( 100 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL',
'log_visit.config_windowsmedia' => 'TINYINT(1) NOT NULL',
'log_visit.config_silverlight' => 'TINYINT(1) NOT NULL',
'log_visit.config_java' => 'TINYINT(1) NOT NULL',
'log_visit.config_gears' => 'TINYINT(1) NOT NULL',
'log_visit.config_pdf' => 'TINYINT(1) NOT NULL',
'log_visit.config_quicktime' => 'TINYINT(1) NOT NULL',
'log_visit.config_realplayer' => 'TINYINT(1) NOT NULL',
'log_visit.config_device_type' => 'TINYINT( 100 ) NULL DEFAULT NULL',
'log_visit.visitor_localtime' => 'TIME NOT NULL',
'log_visit.location_region' => 'char(2) DEFAULT NULL1',
'log_visit.visitor_days_since_last' => 'SMALLINT(5) UNSIGNED NOT NULL',
'log_visit.location_longitude' => 'float(10, 6) DEFAULT NULL1',
'log_visit.visit_total_events' => 'SMALLINT(5) UNSIGNED NOT NULL',
'log_visit.config_os_version' => 'VARCHAR( 100 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL',
'log_visit.location_city' => 'varchar(255) DEFAULT NULL1',
'log_visit.location_country' => 'CHAR(3) NOT NULL1',
'log_visit.location_latitude' => 'float(10, 6) DEFAULT NULL1',
'log_visit.config_flash' => 'TINYINT(1) NOT NULL',
'log_visit.config_director' => 'TINYINT(1) NOT NULL',
'log_visit.visit_total_time' => 'SMALLINT(5) UNSIGNED NOT NULL',
'log_visit.visitor_count_visits' => 'SMALLINT(5) UNSIGNED NOT NULL1',
'log_visit.visit_entry_idaction_name' => 'INTEGER(11) UNSIGNED NOT NULL',
'log_visit.visit_entry_idaction_url' => 'INTEGER(11) UNSIGNED NOT NULL',
'log_visit.visitor_returning' => 'TINYINT(1) NOT NULL1',
'log_visit.visitor_days_since_order' => 'SMALLINT(5) UNSIGNED NOT NULL1',
'log_visit.visit_goal_buyer' => 'TINYINT(1) NOT NULL',
'log_visit.visit_first_action_time' => 'DATETIME NOT NULL',
'log_visit.visit_goal_converted' => 'TINYINT(1) NOT NULL',
'log_visit.visitor_days_since_first' => 'SMALLINT(5) UNSIGNED NOT NULL1',
'log_visit.visit_exit_idaction_name' => 'INTEGER(11) UNSIGNED NOT NULL',
'log_visit.visit_exit_idaction_url' => 'INTEGER(11) UNSIGNED NULL DEFAULT 0',
'log_visit.config_browser_version' => 'VARCHAR(20) NOT NULL',
'log_visit.config_browser_name' => 'VARCHAR(10) NOT NULL',
'log_visit.config_browser_engine' => 'VARCHAR(10) NOT NULL',
'log_visit.location_browser_lang' => 'VARCHAR(20) NOT NULL',
'log_visit.config_os' => 'CHAR(3) NOT NULL',
'log_visit.config_cookie' => 'TINYINT(1) NOT NULL',
'log_visit.referer_url' => 'TEXT NOT NULL',
'log_visit.visit_total_searches' => 'SMALLINT(5) UNSIGNED NOT NULL',
'log_visit.visit_total_actions' => 'SMALLINT(5) UNSIGNED NOT NULL',
'log_visit.referer_keyword' => 'VARCHAR(255) NULL1',
'log_visit.referer_name' => 'VARCHAR(70) NULL1',
'log_visit.referer_type' => 'TINYINT(1) UNSIGNED NULL1',
'log_visit.user_id' => 'VARCHAR(200) NULL',
'log_link_visit_action.idaction_name' => 'INTEGER(10) UNSIGNED',
'log_link_visit_action.idaction_url' => 'INTEGER(10) UNSIGNED DEFAULT NULL',
'log_link_visit_action.server_time' => 'DATETIME NOT NULL',
'log_link_visit_action.time_spent_ref_action' => 'INTEGER(10) UNSIGNED NOT NULL',
'log_link_visit_action.idaction_event_action' => 'INTEGER(10) UNSIGNED DEFAULT NULL',
'log_link_visit_action.idaction_event_category' => 'INTEGER(10) UNSIGNED DEFAULT NULL',
'log_conversion.revenue_discount' => 'float default NULL',
'log_conversion.revenue' => 'float default NULL',
'log_conversion.revenue_shipping' => 'float default NULL',
'log_conversion.revenue_subtotal' => 'float default NULL',
'log_conversion.revenue_tax' => 'float default NULL',
);
if (!array_key_exists($name, $initialCoreDimensionVersions)) {
return false;
}
return strtolower($initialCoreDimensionVersions[$name]) === strtolower($version);
}
public function onNoUpdateAvailable($versionsThatWereChecked)
{
if (!empty($versionsThatWereChecked)) {
// invalidate cache only if there were actually file changes before, otherwise we write the cache on each
// request. There were versions checked only if there was a file change but no update, meaning we can
// set the cache and declare this state as "no update available".
self::cacheCurrentDimensionFileChanges();
}
}
private static function getCurrentDimensionFileChanges()
{
$files = Filesystem::globr(PIWIK_INCLUDE_PATH . '/plugins/*/Columns', '*.php');
$times = array();
foreach ($files as $file) {
$times[$file] = filemtime($file);
}
return $times;
}
private static function cacheCurrentDimensionFileChanges()
{
$changes = self::getCurrentDimensionFileChanges();
$cache = self::buildCache();
$cache->save(self::$cacheId, $changes);
}
private static function buildCache()
{
return PiwikCache::getEagerCache();
}
private static function getCachedDimensionFileChanges()
{
$cache = self::buildCache();
if ($cache->contains(self::$cacheId)) {
return $cache->fetch(self::$cacheId);
}
return array();
}
}

557
www/analytics/core/Common.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,9 +9,11 @@
namespace Piwik;
use Exception;
use Piwik\Container\StaticContainer;
use Piwik\Intl\Data\Provider\LanguageDataProvider;
use Piwik\Intl\Data\Provider\RegionDataProvider;
use Piwik\Plugins\UserCountry\LocationProvider\DefaultProvider;
use Piwik\Tracker;
use Piwik\Tracker\Cache;
use Piwik\Tracker\Cache as TrackerCache;
/**
* Contains helper methods used by both Piwik Core and the Piwik Tracking engine.
@ -34,6 +36,7 @@ class Common
/*
* Database
*/
const LANGUAGE_CODE_INVALID = 'xx';
/**
* Hashes a string into an integer which should be very low collision risks
@ -48,7 +51,7 @@ class Common
/**
* Returns a prefixed table name.
*
*
* The table prefix is determined by the `[database] tables_prefix` INI config
* option.
*
@ -82,7 +85,7 @@ class Common
*
* The table prefix is determined by the `[database] tables_prefix` INI config
* option.
*
*
* @param string $table The prefixed table name, eg "piwik-production_log_visit".
* @return string The unprefixed table name, eg "log_visit".
* @api
@ -107,12 +110,12 @@ class Common
*/
public static function isGoalPluginEnabled()
{
return \Piwik\Plugin\Manager::getInstance()->isPluginActivated('Goals');
return Plugin\Manager::getInstance()->isPluginActivated('Goals');
}
public static function isActionsPluginEnabled()
{
return \Piwik\Plugin\Manager::getInstance()->isPluginActivated('Actions');
return Plugin\Manager::getInstance()->isPluginActivated('Actions');
}
/**
@ -127,21 +130,44 @@ class Common
return self::$isCliMode;
}
$remoteAddr = @$_SERVER['REMOTE_ADDR'];
return PHP_SAPI == 'cli' ||
(!strncmp(PHP_SAPI, 'cgi', 3) && empty($remoteAddr));
if(PHP_SAPI == 'cli'){
return true;
}
if(self::isPhpCgiType() && (!isset($_SERVER['REMOTE_ADDR']) || empty($_SERVER['REMOTE_ADDR']))){
return true;
}
return false;
}
/**
* Returns true if the current request is a console command, eg. ./console xx:yy
* Returns true if PHP is executed as CGI type.
*
* @since added in 0.4.4
* @return bool true if PHP invoked as a CGI
*/
public static function isPhpCgiType()
{
$sapiType = php_sapi_name();
return substr($sapiType, 0, 3) === 'cgi';
}
/**
* Returns true if the current request is a console command, eg.
* ./console xx:yy
* or
* php console xx:yy
*
* @return bool
*/
public static function isRunningConsoleCommand()
{
$searched = '/console';
$searched = 'console';
$consolePos = strpos($_SERVER['SCRIPT_NAME'], $searched);
$expectedConsolePos = strlen($_SERVER['SCRIPT_NAME']) - strlen($searched);
$isScriptIsConsole = $consolePos == $expectedConsolePos;
$isScriptIsConsole = ($consolePos === $expectedConsolePos);
return self::isPhpCliMode() && $isScriptIsConsole;
}
@ -151,7 +177,7 @@ class Common
/**
* Multi-byte substr() - works with UTF-8.
*
*
* Calls `mb_substr` if available and falls back to `substr` if it's not.
*
* @param string $string
@ -175,7 +201,7 @@ class Common
/**
* Multi-byte strlen() - works with UTF-8
*
*
* Calls `mb_substr` if available and falls back to `substr` if not.
*
* @param string $string
@ -193,9 +219,9 @@ class Common
/**
* Multi-byte strtolower() - works with UTF-8.
*
*
* Calls `mb_strtolower` if available and falls back to `strtolower` if not.
*
*
* @param string $string
* @return string
* @api
@ -215,18 +241,18 @@ class Common
/**
* Sanitizes a string to help avoid XSS vulnerabilities.
*
*
* This function is automatically called when {@link getRequestVar()} is called,
* so you should not normally have to use it.
*
*
* This function should be used when outputting data that isn't escaped and was
* obtained from the user (for example when using the `|raw` twig filter on goal names).
*
*
* _NOTE: Sanitized input should not be used directly in an SQL query; SQL placeholders
* should still be used._
*
*
* **Implementation Details**
*
*
* - [htmlspecialchars](http://php.net/manual/en/function.htmlspecialchars.php) is used to escape text.
* - Single quotes are not escaped so **Piwik's amazing community** will still be
* **Piwik's amazing community**.
@ -246,10 +272,11 @@ class Common
if (is_numeric($value)) {
return $value;
} elseif (is_string($value)) {
$value = self::sanitizeInputValue($value);
$value = self::sanitizeString($value);
if (!$alreadyStripslashed) {
// a JSON array was already stripslashed, don't do it again for each value
if (!$alreadyStripslashed) // a JSON array was already stripslashed, don't do it again for each value
{
$value = self::undoMagicQuotes($value);
}
} elseif (is_array($value)) {
@ -272,20 +299,32 @@ class Common
}
/**
* Sanitize a single input value
* Sanitize a single input value and removes line breaks, tabs and null characters.
*
* @param string $value
* @return string sanitized input
*/
public static function sanitizeInputValue($value)
{
$value = self::sanitizeLineBreaks($value);
$value = self::sanitizeString($value);
return $value;
}
/**
* Sanitize a single input value
*
* @param $value
* @return string
*/
private static function sanitizeString($value)
{
// $_GET and $_REQUEST already urldecode()'d
// decode
// note: before php 5.2.7, htmlspecialchars() double encodes &#x hex items
$value = html_entity_decode($value, self::HTML_ENCODING_QUOTE_STYLE, 'UTF-8');
// filter
$value = self::sanitizeLineBreaks($value);
$value = self::sanitizeNullBytes($value);
// escape
$tmp = @htmlspecialchars($value, self::HTML_ENCODING_QUOTE_STYLE, 'UTF-8');
@ -295,15 +334,17 @@ class Common
// convert and escape
$value = utf8_encode($value);
$tmp = htmlspecialchars($value, self::HTML_ENCODING_QUOTE_STYLE, 'UTF-8');
return $tmp;
}
return $tmp;
}
/**
* Unsanitizes a single input value and returns the result.
*
*
* @param string $value
* @return string unsanitized input
* @api
*/
public static function unsanitizeInputValue($value)
{
@ -315,10 +356,10 @@ class Common
*
* This method should be used when you need to unescape data that was obtained from
* the user.
*
*
* Some data in Piwik is stored sanitized (such as site name). In this case you may
* have to use this method to unsanitize it in order to, for example, output it in JSON.
*
*
* @param string|array $value The data to unsanitize. If an array is passed, the
* array is sanitized recursively. Key values are not unsanitized.
* @return string|array The unsanitized data.
@ -345,28 +386,42 @@ class Common
*/
private static function undoMagicQuotes($value)
{
return version_compare(PHP_VERSION, '5.4', '<')
&& get_magic_quotes_gpc()
? stripslashes($value)
: $value;
}
static $shouldUndo;
if (!isset($shouldUndo)) {
$shouldUndo = version_compare(PHP_VERSION, '5.4', '<') && get_magic_quotes_gpc();
}
if ($shouldUndo) {
$value = stripslashes($value);
}
/**
*
* @param string
* @return string Line breaks and line carriage removed
*/
public static function sanitizeLineBreaks($value)
{
$value = str_replace(array("\n", "\r", "\0"), '', $value);
return $value;
}
/**
* @param string $value
* @return string Line breaks and line carriage removed
*/
public static function sanitizeLineBreaks($value)
{
return str_replace(array("\n", "\r"), '', $value);
}
/**
* @param string $value
* @return string Null bytes removed
*/
public static function sanitizeNullBytes($value)
{
return str_replace(array("\0"), '', $value);
}
/**
* Gets a sanitized request parameter by name from the `$_GET` and `$_POST` superglobals.
*
*
* Use this function to get request parameter values. **_NEVER use `$_GET` and `$_POST` directly._**
*
*
* If the variable cannot be found, and a default value was not provided, an exception is raised.
*
* _See {@link sanitizeInputValues()} to learn more about sanitization._
@ -376,7 +431,7 @@ class Common
* @param string|null $varDefault The value to return if the request parameter cannot be found or has an empty value.
* @param string|null $varType Expected type of the request variable. This parameters value must be one of the following:
* `'array'`, `'int'`, `'integer'`, `'string'`, `'json'`.
*
*
* If `'json'`, the string value will be `json_decode`-d and then sanitized.
* @param array|null $requestArrayToUse The array to use instead of `$_GET` and `$_POST`.
* @throws Exception If the request parameter doesn't exist and there is no default value, or if the request parameter
@ -389,6 +444,7 @@ class Common
if (is_null($requestArrayToUse)) {
$requestArrayToUse = $_GET + $_POST;
}
$varDefault = self::sanitizeInputValues($varDefault);
if ($varType === 'int') {
// settype accepts only integer
@ -420,22 +476,36 @@ class Common
// we deal w/ json differently
if ($varType == 'json') {
$value = self::undoMagicQuotes($requestArrayToUse[$varName]);
$value = self::json_decode($value, $assoc = true);
$value = json_decode($value, $assoc = true);
return self::sanitizeInputValues($value, $alreadyStripslashed = true);
}
$value = self::sanitizeInputValues($requestArrayToUse[$varName]);
if (!is_null($varType)) {
if (isset($varType)) {
$ok = false;
if ($varType === 'string') {
if (is_string($value)) $ok = true;
if (is_string($value) || is_int($value)) {
$ok = true;
} elseif (is_float($value)) {
$value = Common::forceDotAsSeparatorForDecimalPoint($value);
$ok = true;
}
} elseif ($varType === 'integer') {
if ($value == (string)(int)$value) $ok = true;
if ($value == (string)(int)$value) {
$ok = true;
}
} elseif ($varType === 'float') {
if ($value == (string)(float)$value) $ok = true;
$valueToCompare = (string)(float)$value;
$valueToCompare = Common::forceDotAsSeparatorForDecimalPoint($valueToCompare);
if ($value == $valueToCompare) {
$ok = true;
}
} elseif ($varType === 'array') {
if (is_array($value)) $ok = true;
if (is_array($value)) {
$ok = true;
}
} else {
throw new Exception("\$varType specified is not known. It should be one of the following: array, int, integer, float, string");
}
@ -452,6 +522,7 @@ class Common
}
settype($value, $varType);
}
return $value;
}
@ -466,11 +537,17 @@ class Common
*/
public static function generateUniqId()
{
return md5(uniqid(rand(), true));
if (function_exists('mt_rand')) {
$rand = mt_rand();
} else {
$rand = rand();
}
return md5(uniqid($rand, true));
}
/**
* Configureable hash() algorithm (defaults to md5)
* Configurable hash() algorithm (defaults to md5)
*
* @param string $str String to be hashed
* @param bool $raw_output
@ -479,14 +556,16 @@ class Common
public static function hash($str, $raw_output = false)
{
static $hashAlgorithm = null;
if (is_null($hashAlgorithm)) {
$hashAlgorithm = @Config::getInstance()->General['hash_algorithm'];
}
if ($hashAlgorithm) {
$hash = @hash($hashAlgorithm, $str, $raw_output);
if ($hash !== false)
if ($hash !== false) {
return $hash;
}
}
return md5($str, $raw_output);
@ -503,16 +582,13 @@ class Common
public static function getRandomString($length = 16, $alphabet = "abcdefghijklmnoprstuvwxyz0123456789")
{
$chars = $alphabet;
$str = '';
list($usec, $sec) = explode(" ", microtime());
$seed = ((float)$sec + (float)$usec) * 100000;
mt_srand($seed);
$str = '';
for ($i = 0; $i < $length; $i++) {
$rand_key = mt_rand(0, strlen($chars) - 1);
$str .= substr($chars, $rand_key, 1);
}
return str_shuffle($str);
}
@ -554,27 +630,22 @@ class Common
) {
throw new Exception("visitorId is expected to be a " . Tracker::LENGTH_HEX_ID_STRING . " hex char string");
}
return self::hex2bin($id);
}
/**
* Convert IP address (in network address format) to presentation format.
* This is a backward compatibility function for code that only expects
* IPv4 addresses (i.e., doesn't support IPv6).
* Converts a User ID string to the Visitor ID Binary representation.
*
* @see IP::N2P()
*
* This function does not support the long (or its string representation)
* returned by the built-in ip2long() function, from Piwik 1.3 and earlier.
*
* @deprecated 1.4
*
* @param string $ip IP address in network address format
* @param $userId
* @return string
*/
public static function long2ip($ip)
public static function convertUserIdToVisitorIdBin($userId)
{
return IP::long2ip($ip);
require_once PIWIK_INCLUDE_PATH . '/libs/PiwikTracker/PiwikTracker.php';
$userIdHashed = \PiwikTracker::getUserIdHashed($userId);
return self::convertVisitorIdToBin($userIdHashed);
}
/**
@ -653,14 +724,14 @@ class Common
/**
* Returns the list of parent classes for the given class.
*
* @param string $klass A class name.
* @param string $class A class name.
* @return string[] The list of parent classes in order from highest ancestor to the descended class.
*/
public static function getClassLineage($klass)
public static function getClassLineage($class)
{
$klasses = array_merge(array($klass), array_values(class_parents($klass, $autoload = false)));
$classes = array_merge(array($class), array_values(class_parents($class, $autoload = false)));
return array_reverse($klasses);
return array_reverse($classes);
}
/*
@ -672,14 +743,16 @@ class Common
*
* @see core/DataFiles/Countries.php
*
* @return array Array of 3 letter continent codes
* @return array Array of 3 letter continent codes
*
* @deprecated Use Piwik\Intl\Data\Provider\RegionDataProvider instead.
* @see \Piwik\Intl\Data\Provider\RegionDataProvider::getContinentList()
*/
public static function getContinentsList()
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/Countries.php';
$continentsList = $GLOBALS['Piwik_ContinentList'];
return $continentsList;
/** @var RegionDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\RegionDataProvider');
return $dataProvider->getContinentList();
}
/**
@ -688,19 +761,16 @@ class Common
* @see core/DataFiles/Countries.php
*
* @param bool $includeInternalCodes
* @return array Array of (2 letter ISO codes => 3 letter continent code)
* @return array Array of (2 letter ISO codes => 3 letter continent code)
*
* @deprecated Use Piwik\Intl\Data\Provider\RegionDataProvider instead.
* @see \Piwik\Intl\Data\Provider\RegionDataProvider::getCountryList()
*/
public static function getCountriesList($includeInternalCodes = false)
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/Countries.php';
$countriesList = $GLOBALS['Piwik_CountryList'];
$extras = $GLOBALS['Piwik_CountryList_Extras'];
if ($includeInternalCodes) {
return array_merge($countriesList, $extras);
}
return $countriesList;
/** @var RegionDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\RegionDataProvider');
return $dataProvider->getCountryList($includeInternalCodes);
}
/**
@ -711,13 +781,15 @@ class Common
* @return array Array of two letter ISO codes mapped with their associated language names (in English). E.g.
* `array('en' => 'English', 'ja' => 'Japanese')`.
* @api
*
* @deprecated Use Piwik\Intl\Data\Provider\LanguageDataProvider instead.
* @see \Piwik\Intl\Data\Provider\LanguageDataProvider::getLanguageList()
*/
public static function getLanguagesList()
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/Languages.php';
$languagesList = $GLOBALS['Piwik_LanguageList'];
return $languagesList;
/** @var LanguageDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\LanguageDataProvider');
return $dataProvider->getLanguageList();
}
/**
@ -728,70 +800,15 @@ class Common
* @return array Array of two letter ISO language codes mapped with two letter ISO country codes:
* `array('fr' => 'fr') // French => France`
* @api
*
* @deprecated Use Piwik\Intl\Data\Provider\LanguageDataProvider instead.
* @see \Piwik\Intl\Data\Provider\LanguageDataProvider::getLanguageToCountryList()
*/
public static function getLanguageToCountryList()
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/LanguageToCountry.php';
$languagesList = $GLOBALS['Piwik_LanguageToCountry'];
return $languagesList;
}
/**
* Returns list of search engines by URL
*
* @see core/DataFiles/SearchEngines.php
*
* @return array Array of ( URL => array( searchEngineName, keywordParameter, path, charset ) )
*/
public static function getSearchEngineUrls()
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/SearchEngines.php';
$searchEngines = $GLOBALS['Piwik_SearchEngines'];
Piwik::postEvent('Referrer.addSearchEngineUrls', array(&$searchEngines));
return $searchEngines;
}
/**
* Returns list of search engines by name
*
* @see core/DataFiles/SearchEngines.php
*
* @return array Array of ( searchEngineName => URL )
*/
public static function getSearchEngineNames()
{
$searchEngines = self::getSearchEngineUrls();
$nameToUrl = array();
foreach ($searchEngines as $url => $info) {
if (!isset($nameToUrl[$info[0]])) {
$nameToUrl[$info[0]] = $url;
}
}
return $nameToUrl;
}
/**
* Returns list of social networks by URL
*
* @see core/DataFiles/Socials.php
*
* @return array Array of ( URL => Social Network Name )
*/
public static function getSocialUrls()
{
require_once PIWIK_INCLUDE_PATH . '/core/DataFiles/Socials.php';
$socialUrls = $GLOBALS['Piwik_socialUrl'];
Piwik::postEvent('Referrer.addSocialUrls', array(&$socialUrls));
return $socialUrls;
/** @var LanguageDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\LanguageDataProvider');
return $dataProvider->getLanguageToCountryList();
}
/**
@ -840,7 +857,7 @@ class Common
}
}
if (is_null($browserLang)) {
if (empty($browserLang)) {
// a fallback might be to infer the language in HTTP_USER_AGENT (i.e., localized build)
$browserLang = "";
} else {
@ -872,11 +889,15 @@ class Common
*/
public static function getCountry($lang, $enableLanguageToCountryGuess, $ip)
{
if (empty($lang) || strlen($lang) < 2 || $lang == 'xx') {
return 'xx';
if (empty($lang) || strlen($lang) < 2 || $lang == self::LANGUAGE_CODE_INVALID) {
return self::LANGUAGE_CODE_INVALID;
}
$validCountries = self::getCountriesList();
/** @var RegionDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\RegionDataProvider');
$validCountries = $dataProvider->getCountryList();
return self::extractCountryCodeFromBrowserLanguage($lang, $validCountries, $enableLanguageToCountryGuess);
}
@ -890,7 +911,10 @@ class Common
*/
public static function extractCountryCodeFromBrowserLanguage($browserLanguage, $validCountries, $enableLanguageToCountryGuess)
{
$langToCountry = self::getLanguageToCountryList();
/** @var LanguageDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\LanguageDataProvider');
$langToCountry = $dataProvider->getLanguageToCountryList();
if ($enableLanguageToCountryGuess) {
if (preg_match('/^([a-z]{2,3})(?:,|;|$)/', $browserLanguage, $matches)) {
@ -909,51 +933,90 @@ class Common
}
}
}
return 'xx';
return self::LANGUAGE_CODE_INVALID;
}
/**
* Returns the visitor language based only on the Browser 'accepted language' information
* Returns the language and region string, based only on the Browser 'accepted language' information.
* * The language tag is defined by ISO 639-1
*
* @param string $browserLanguage Browser's accepted langauge header
* @param array $validLanguages array of valid language codes
* @return string 2 letter ISO 639 code
* @return string 2 letter ISO 639 code 'es' (Spanish)
*/
public static function extractLanguageCodeFromBrowserLanguage($browserLanguage, $validLanguages)
public static function extractLanguageCodeFromBrowserLanguage($browserLanguage, $validLanguages = array())
{
// assumes language preference is sorted;
// does not handle language-script-region tags or language range (*)
if (!empty($validLanguages) && preg_match_all('/(?:^|,)([a-z]{2,3})([-][a-z]{2})?/', $browserLanguage, $matches, PREG_SET_ORDER)) {
foreach ($matches as $parts) {
if (count($parts) == 3) {
// match locale (language and location)
if (in_array($parts[1] . $parts[2], $validLanguages)) {
return $parts[1] . $parts[2];
}
$validLanguages = self::checkValidLanguagesIsSet($validLanguages);
$languageRegionCode = self::extractLanguageAndRegionCodeFromBrowserLanguage($browserLanguage, $validLanguages);
if (strlen($languageRegionCode) == 2) {
$languageCode = $languageRegionCode;
} else {
$languageCode = substr($languageRegionCode, 0, 2);
}
if (in_array($languageCode, $validLanguages)) {
return $languageCode;
}
return self::LANGUAGE_CODE_INVALID;
}
/**
* Returns the language and region string, based only on the Browser 'accepted language' information.
* * The language tag is defined by ISO 639-1
* * The region tag is defined by ISO 3166-1
*
* @param string $browserLanguage Browser's accepted langauge header
* @param array $validLanguages array of valid language codes. Note that if the array includes "fr" then it will consider all regional variants of this language valid, such as "fr-ca" etc.
* @return string 2 letter ISO 639 code 'es' (Spanish) or if found, includes the region as well: 'es-ar'
*/
public static function extractLanguageAndRegionCodeFromBrowserLanguage($browserLanguage, $validLanguages = array())
{
$validLanguages = self::checkValidLanguagesIsSet($validLanguages);
if (!preg_match_all('/(?:^|,)([a-z]{2,3})([-][a-z]{2})?/', $browserLanguage, $matches, PREG_SET_ORDER)) {
return self::LANGUAGE_CODE_INVALID;
}
foreach ($matches as $parts) {
$langIso639 = $parts[1];
if (empty($langIso639)) {
continue;
}
// If a region tag is found eg. "fr-ca"
if (count($parts) == 3) {
$regionIso3166 = $parts[2]; // eg. "-ca"
if (in_array($langIso639 . $regionIso3166, $validLanguages)) {
return $langIso639 . $regionIso3166;
}
// match language only (where no region provided)
if (in_array($parts[1], $validLanguages)) {
return $parts[1];
if (in_array($langIso639, $validLanguages)) {
return $langIso639 . $regionIso3166;
}
}
// eg. "fr" or "es"
if (in_array($langIso639, $validLanguages)) {
return $langIso639;
}
}
return 'xx';
return self::LANGUAGE_CODE_INVALID;
}
/**
* Returns the continent of a given country
*
* @param string $country 2 letters isocode
* @param string $country 2 letters iso code
*
* @return string Continent (3 letters code : afr, asi, eur, amn, ams, oce)
*/
public static function getContinent($country)
{
$countryList = self::getCountriesList();
if (isset($countryList[$country])) {
return $countryList[$country];
}
return 'unk';
/** @var RegionDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\RegionDataProvider');
$countryList = $dataProvider->getCountryList();
return isset($countryList[$country]) ? $countryList[$country] : 'unk';
}
/*
@ -995,10 +1058,10 @@ class Common
/**
* Returns a string with a comma separated list of placeholders for use in an SQL query. Used mainly
* to fill the `IN (...)` part of a query.
*
*
* @param array|string $fields The names of the mysql table fields to bind, e.g.
* `array(fieldName1, fieldName2, fieldName3)`.
*
*
* _Note: The content of the array isn't important, just its length._
* @return string The placeholder string, e.g. `"?, ?, ?"`.
* @api
@ -1015,6 +1078,22 @@ class Common
return '?' . str_repeat(',?', $count - 1);
}
/**
* Force the separator for decimal point to be a dot. See https://github.com/piwik/piwik/issues/6435
* If for instance a German locale is used it would be a comma otherwise.
*
* @param float|string $value
* @return string
*/
public static function forceDotAsSeparatorForDecimalPoint($value)
{
if (null === $value || false === $value) {
return $value;
}
return str_replace(',', '.', $value);
}
/**
* Sets outgoing header.
*
@ -1024,23 +1103,62 @@ class Common
public static function sendHeader($header, $replace = true)
{
// don't send header in CLI mode
if(Common::isPhpCliMode()) {
return;
}
if (isset($GLOBALS['PIWIK_TRACKER_LOCAL_TRACKING']) && $GLOBALS['PIWIK_TRACKER_LOCAL_TRACKING']) {
@header($header, $replace);
} else {
if (!Common::isPhpCliMode() and !headers_sent()) {
header($header, $replace);
}
}
/**
* Sends the given response code if supported.
*
* @param int $code Eg 204
*
* @throws Exception
*/
public static function sendResponseCode($code)
{
$messages = array(
200 => 'Ok',
204 => 'No Response',
301 => 'Moved Permanently',
302 => 'Found',
304 => 'Not Modified',
400 => 'Bad Request',
401 => 'Unauthorized',
403 => 'Forbidden',
404 => 'Not Found',
500 => 'Internal Server Error',
503 => 'Service Unavailable',
);
if (!array_key_exists($code, $messages)) {
throw new Exception('Response code not supported: ' . $code);
}
if (strpos(PHP_SAPI, '-fcgi') === false) {
$key = 'HTTP/1.1';
if (array_key_exists('SERVER_PROTOCOL', $_SERVER)
&& strlen($_SERVER['SERVER_PROTOCOL']) < 15
&& strlen($_SERVER['SERVER_PROTOCOL']) > 1) {
$key = $_SERVER['SERVER_PROTOCOL'];
}
} else {
// FastCGI
$key = 'Status:';
}
$message = $messages[$code];
Common::sendHeader($key . ' ' . $code . ' ' . $message);
}
/**
* Returns the ID of the current LocationProvider (see UserCountry plugin code) from
* the Tracker cache.
*/
public static function getCurrentLocationProviderId()
{
$cache = Cache::getCacheGeneral();
$cache = TrackerCache::getCacheGeneral();
return empty($cache['currentLocationProviderId'])
? DefaultProvider::ID
: $cache['currentLocationProviderId'];
@ -1049,11 +1167,11 @@ class Common
/**
* Marks an orphaned object for garbage collection.
*
* For more information: {@link http://dev.piwik.org/trac/ticket/374}
* @param $var The object to destroy.
* For more information: {@link https://github.com/piwik/piwik/issues/374}
* @param mixed $var The object to destroy.
* @api
*/
static public function destroy(&$var)
public static function destroy(&$var)
{
if (is_object($var) && method_exists($var, '__destruct')) {
$var->__destruct();
@ -1062,27 +1180,62 @@ class Common
$var = null;
}
static public function printDebug($info = '')
/**
* @todo This method is weird, it's debugging statements but seem to only work for the tracker, maybe it
* should be moved elsewhere
*/
public static function printDebug($info = '')
{
if (isset($GLOBALS['PIWIK_TRACKER_DEBUG']) && $GLOBALS['PIWIK_TRACKER_DEBUG']) {
if (!headers_sent()) {
// prevent XSS in tracker debug output
header('Content-type: text/plain');
}
if (is_object($info)) {
$info = var_export($info, true);
}
Log::getInstance()->setLogLevel(Log::DEBUG);
$logger = StaticContainer::get('Psr\Log\LoggerInterface');
if (is_array($info) || is_object($info)) {
$info = Common::sanitizeInputValues($info);
$out = var_export($info, true);
foreach (explode("\n", $out) as $line) {
Log::debug($line);
$logger->debug($line);
}
} else {
foreach (explode("\n", $info) as $line) {
Log::debug(htmlspecialchars($line, ENT_QUOTES));
$logger->debug($line);
}
}
}
}
/**
* Returns true if the request is an AJAX request.
*
* @return bool
*/
public static function isXmlHttpRequest()
{
return isset($_SERVER['HTTP_X_REQUESTED_WITH'])
&& (strtolower($_SERVER['HTTP_X_REQUESTED_WITH']) === 'xmlhttprequest');
}
/**
* @param $validLanguages
* @return array
*/
protected static function checkValidLanguagesIsSet($validLanguages)
{
/** @var LanguageDataProvider $dataProvider */
$dataProvider = StaticContainer::get('Piwik\Intl\Data\Provider\LanguageDataProvider');
if (empty($validLanguages)) {
$validLanguages = array_keys($dataProvider->getLanguageList());
return $validLanguages;
}
return $validLanguages;
}
}

47
www/analytics/core/Composer/ScriptHandler.php Normal file
View file

@ -0,0 +1,47 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Composer;
/**
* Scripts executed before/after Composer install and update.
*
* We use this PHP class because setting the bash scripts directly in composer.json breaks
* Composer on Windows systems.
*/
class ScriptHandler
{
private static function isPhp7orLater()
{
return version_compare('7.0.0-dev', PHP_VERSION) < 1;
}
public static function cleanXhprof()
{
if (! is_dir('vendor/facebook/xhprof/extension')) {
return;
}
if (!self::isPhp7orLater()) {
// doesn't work with PHP 7 at the moment
passthru('misc/composer/clean-xhprof.sh');
}
}
public static function buildXhprof()
{
if (! is_dir('vendor/facebook/xhprof/extension')) {
return;
}
if (!self::isPhp7orLater()) {
passthru('misc/composer/clean-xhprof.sh');
}
}
}

170
www/analytics/core/Concurrency/DistributedList.php Normal file
View file

@ -0,0 +1,170 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Concurrency;
use Piwik\Container\StaticContainer;
use Piwik\Option;
use Psr\Log\LoggerInterface;
/**
* Manages a simple distributed list stored in an Option. No locking occurs, so the list
* is not thread safe, and should only be used for use cases where atomicity is not
* important.
*
* The list of items is serialized and stored in an Option. Items are converted to string
* before being persisted, so it is not expected to unserialize objects.
*/
class DistributedList
{
/**
* The name of the option to store the list in.
*
* @var string
*/
private $optionName;
/**
* @var LoggerInterface
*/
private $logger;
/**
* Constructor.
*
* @param string $optionName
*/
public function __construct($optionName, LoggerInterface $logger = null)
{
$this->optionName = $optionName;
$this->logger = $logger ?: StaticContainer::get('Psr\Log\LoggerInterface');
}
/**
* Queries the option table and returns all items in this list.
*
* @return array
*/
public function getAll()
{
$result = $this->getListOptionValue();
foreach ($result as $key => $item) {
// remove non-array items (unexpected state, though can happen when upgrading from an old Piwik)
if (is_array($item)) {
$this->logger->info("Found array item in DistributedList option value '{name}': {data}", array(
'name' => $this->optionName,
'data' => var_export($result, true)
));
unset($result[$key]);
}
}
return $result;
}
/**
* Sets the contents of the list in the option table.
*
* @param string[] $items
*/
public function setAll($items)
{
foreach ($items as $key => &$item) {
if (is_array($item)) {
throw new \InvalidArgumentException("Array item encountered in DistributedList::setAll() [ key = $key ].");
} else {
$item = (string)$item;
}
}
Option::set($this->optionName, serialize($items));
}
/**
* Adds one or more items to the list in the option table.
*
* @param string|array $item
*/
public function add($item)
{
$allItems = $this->getAll();
if (is_array($item)) {
$allItems = array_merge($allItems, $item);
} else {
$allItems[] = $item;
}
$this->setAll($allItems);
}
/**
* Removes one or more items by value from the list in the option table.
*
* Does not preserve array keys.
*
* @param string|array $items
*/
public function remove($items)
{
if (!is_array($items)) {
$items = array($items);
}
$allItems = $this->getAll();
foreach ($items as $item) {
$existingIndex = array_search($item, $allItems);
if ($existingIndex === false) {
return;
}
unset($allItems[$existingIndex]);
}
$this->setAll(array_values($allItems));
}
/**
* Removes one or more items by index from the list in the option table.
*
* Does not preserve array keys.
*
* @param int[]|int $indices
*/
public function removeByIndex($indices)
{
if (!is_array($indices)) {
$indices = array($indices);
}
$indices = array_unique($indices);
$allItems = $this->getAll();
foreach ($indices as $index) {
unset($allItems[$index]);
}
$this->setAll(array_values($allItems));
}
protected function getListOptionValue()
{
Option::clearCachedOption($this->optionName);
$array = Option::get($this->optionName);
$result = array();
if ($array
&& ($array = unserialize($array))
&& count($array)
) {
$result = $array;
}
return $result;
}
}

561
www/analytics/core/Config.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -10,16 +10,18 @@
namespace Piwik;
use Exception;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Container\StaticContainer;
/**
* Singleton that provides read & write access to Piwik's INI configuration.
*
*
* This class reads and writes to the `config/config.ini.php` file. If config
* options are missing from that file, this class will look for their default
* values in `config/global.ini.php`.
*
*
* ### Examples
*
*
* **Getting a value:**
*
* // read the minimum_memory_limit option under the [General] section
@ -30,47 +32,39 @@ use Exception;
* // set the minimum_memory_limit option
* Config::getInstance()->General['minimum_memory_limit'] = 256;
* Config::getInstance()->forceSave();
*
*
* **Setting an entire section:**
*
*
* Config::getInstance()->MySection = array('myoption' => 1);
* Config::getInstance()->forceSave();
*
* @method static \Piwik\Config getInstance()
*/
class Config extends Singleton
class Config
{
const DEFAULT_LOCAL_CONFIG_PATH = '/config/config.ini.php';
const DEFAULT_COMMON_CONFIG_PATH = '/config/common.config.ini.php';
const DEFAULT_GLOBAL_CONFIG_PATH = '/config/global.ini.php';
/**
* Contains configuration files values
*
* @var array
*/
protected $initialized = false;
protected $configGlobal = array();
protected $configLocal = array();
protected $configCommon = array();
protected $configCache = array();
protected $pathGlobal = null;
protected $pathCommon = null;
protected $pathLocal = null;
/**
* @var boolean
*/
protected $isTest = false;
protected $doNotWriteConfigInTests = false;
/**
* Constructor
* @var GlobalSettingsProvider
*/
public function __construct($pathGlobal = null, $pathLocal = null, $pathCommon = null)
protected $settings;
/**
* @return Config
*/
public static function getInstance()
{
$this->pathGlobal = $pathGlobal ?: self::getGlobalConfigPath();
$this->pathCommon = $pathCommon ?: self::getCommonConfigPath();
$this->pathLocal = $pathLocal ?: self::getLocalConfigPath();
return StaticContainer::get('Piwik\Config');
}
public function __construct(GlobalSettingsProvider $settings)
{
$this->settings = $settings;
}
/**
@ -80,7 +74,7 @@ class Config extends Singleton
*/
public function getLocalPath()
{
return $this->pathLocal;
return $this->settings->getPathLocal();
}
/**
@ -90,7 +84,7 @@ class Config extends Singleton
*/
public function getGlobalPath()
{
return $this->pathGlobal;
return $this->settings->getPathGlobal();
}
/**
@ -100,72 +94,7 @@ class Config extends Singleton
*/
public function getCommonPath()
{
return $this->pathCommon;
}
/**
* Enable test environment
*
* @param string $pathLocal
* @param string $pathGlobal
* @param string $pathCommon
*/
public function setTestEnvironment($pathLocal = null, $pathGlobal = null, $pathCommon = null, $allowSaving = false)
{
if (!$allowSaving) {
$this->isTest = true;
}
$this->clear();
$this->pathLocal = $pathLocal ?: Config::getLocalConfigPath();
$this->pathGlobal = $pathGlobal ?: Config::getGlobalConfigPath();
$this->pathCommon = $pathCommon ?: Config::getCommonConfigPath();
$this->init();
// this proxy will not record any data in the production database.
// this provides security for Piwik installs and tests were setup.
if (isset($this->configGlobal['database_tests'])
|| isset($this->configLocal['database_tests'])
) {
$this->__get('database_tests');
$this->configCache['database'] = $this->configCache['database_tests'];
}
// Ensure local mods do not affect tests
if (empty($pathGlobal)) {
$this->configCache['log'] = $this->configGlobal['log'];
$this->configCache['Debug'] = $this->configGlobal['Debug'];
$this->configCache['mail'] = $this->configGlobal['mail'];
$this->configCache['General'] = $this->configGlobal['General'];
$this->configCache['Segments'] = $this->configGlobal['Segments'];
$this->configCache['Tracker'] = $this->configGlobal['Tracker'];
$this->configCache['Deletelogs'] = $this->configGlobal['Deletelogs'];
$this->configCache['Deletereports'] = $this->configGlobal['Deletereports'];
}
// for unit tests, we set that no plugin is installed. This will force
// the test initialization to create the plugins tables, execute ALTER queries, etc.
$this->configCache['PluginsInstalled'] = array('PluginsInstalled' => array());
// DevicesDetection plugin is not yet enabled by default
if (isset($configGlobal['Plugins'])) {
$this->configCache['Plugins'] = $this->configGlobal['Plugins'];
$this->configCache['Plugins']['Plugins'][] = 'DevicesDetection';
}
if (isset($configGlobal['Plugins_Tracker'])) {
$this->configCache['Plugins_Tracker'] = $this->configGlobal['Plugins_Tracker'];
$this->configCache['Plugins_Tracker']['Plugins_Tracker'][] = 'DevicesDetection';
}
// to avoid weird session error in travis
if (empty($pathGlobal)) {
$configArray = &$this->configCache;
} else {
$configArray = &$this->configLocal;
}
$configArray['General']['session_save_handler'] = 'dbtables';
return $this->settings->getPathCommon();
}
/**
@ -173,7 +102,7 @@ class Config extends Singleton
*
* @return string
*/
protected static function getGlobalConfigPath()
public static function getGlobalConfigPath()
{
return PIWIK_USER_PATH . self::DEFAULT_GLOBAL_CONFIG_PATH;
}
@ -204,6 +133,8 @@ class Config extends Singleton
private static function getLocalConfigInfoForHostname($hostname)
{
// Remove any port number to get actual hostname
$hostname = Url::getHostSanitized($hostname);
$perHostFilename = $hostname . '.config.ini.php';
$pathDomainConfig = PIWIK_USER_PATH . '/config/' . $perHostFilename;
@ -225,11 +156,25 @@ class Config extends Singleton
return array(
'action_url_category_delimiter' => $general['action_url_category_delimiter'],
'autocomplete_min_sites' => $general['autocomplete_min_sites'],
'datatable_export_range_as_day' => $general['datatable_export_range_as_day']
'datatable_export_range_as_day' => $general['datatable_export_range_as_day'],
'datatable_row_limits' => $this->getDatatableRowLimits(),
'are_ads_enabled' => $general['piwik_pro_ads_enabled']
);
}
protected static function getByDomainConfigPath()
/**
* @param $general
* @return mixed
*/
private function getDatatableRowLimits()
{
$limits = $this->General['datatable_row_limits'];
$limits = explode(",", $limits);
$limits = array_map('trim', $limits);
return $limits;
}
public static function getByDomainConfigPath()
{
$host = self::getHostname();
$hostConfig = self::getLocalConfigInfoForHostname($host);
@ -242,9 +187,19 @@ class Config extends Singleton
return false;
}
protected static function getHostname()
/**
* Returns the hostname of the current request (without port number)
*
* @return string
*/
public static function getHostname()
{
$host = Url::getHost($checkIfTrusted = false); // Check trusted requires config file which is not ready yet
// Check trusted requires config file which is not ready yet
$host = Url::getHost($checkIfTrusted = false);
// Remove any port number to get actual hostname
$host = Url::getHostSanitized($host);
return $host;
}
@ -259,19 +214,26 @@ class Config extends Singleton
* @param string $hostname eg piwik.example.com
* @return string
* @throws \Exception In case the domain contains not allowed characters
* @internal
*/
public function forceUsageOfLocalHostnameConfig($hostname)
{
$hostConfig = static::getLocalConfigInfoForHostname($hostname);
$hostConfig = self::getLocalConfigInfoForHostname($hostname);
if (!Filesystem::isValidFilename($hostConfig['file'])) {
throw new Exception('Hostname is not valid');
$filename = $hostConfig['file'];
if (!Filesystem::isValidFilename($filename)) {
throw new Exception('Piwik domain is not a valid looking hostname (' . $filename . ').');
}
$this->pathLocal = $hostConfig['path'];
$this->configLocal = array();
$this->initialized = false;
return $this->pathLocal;
$pathLocal = $hostConfig['path'];
try {
$this->reload($pathLocal);
} catch (Exception $ex) {
// pass (not required for local file to exist at this point)
}
return $pathLocal;
}
/**
@ -281,99 +243,57 @@ class Config extends Singleton
*/
public function isFileWritable()
{
return is_writable($this->pathLocal);
return is_writable($this->settings->getPathLocal());
}
/**
* Clear in-memory configuration so it can be reloaded
* @deprecated since v2.12.0
*/
public function clear()
{
$this->configGlobal = array();
$this->configLocal = array();
$this->configCache = array();
$this->initialized = false;
$this->reload();
}
/**
* Read configuration from files into memory
*
* @throws Exception if local config file is not readable; exits for other errors
* @deprecated since v2.12.0
*/
public function init()
{
$this->initialized = true;
$reportError = SettingsServer::isTrackerApiRequest();
// read defaults from global.ini.php
if (!is_readable($this->pathGlobal) && $reportError) {
Piwik_ExitWithMessage(Piwik::translate('General_ExceptionConfigurationFileNotFound', array($this->pathGlobal)));
}
$this->configGlobal = _parse_ini_file($this->pathGlobal, true);
if (empty($this->configGlobal) && $reportError) {
Piwik_ExitWithMessage(Piwik::translate('General_ExceptionUnreadableFileDisabledMethod', array($this->pathGlobal, "parse_ini_file()")));
}
$this->configCommon = _parse_ini_file($this->pathCommon, true);
// Check config.ini.php last
$this->checkLocalConfigFound();
$this->configLocal = _parse_ini_file($this->pathLocal, true);
if (empty($this->configLocal) && $reportError) {
Piwik_ExitWithMessage(Piwik::translate('General_ExceptionUnreadableFileDisabledMethod', array($this->pathLocal, "parse_ini_file()")));
}
$this->reload();
}
/**
* Reloads config data from disk.
*
* @throws \Exception if the global config file is not found and this is a tracker request, or
* if the local config file is not found and this is NOT a tracker request.
*/
protected function reload($pathLocal = null, $pathGlobal = null, $pathCommon = null)
{
$this->settings->reload($pathGlobal, $pathLocal, $pathCommon);
}
/**
* @deprecated
*/
public function existsLocalConfig()
{
return is_readable($this->pathLocal);
return is_readable($this->getLocalPath());
}
public function checkLocalConfigFound()
public function deleteLocalConfig()
{
if (!$this->existsLocalConfig()) {
throw new Exception(Piwik::translate('General_ExceptionConfigurationFileNotFound', array($this->pathLocal)));
$configLocal = $this->getLocalPath();
if(file_exists($configLocal)){
@unlink($configLocal);
}
}
/**
* Decode HTML entities
*
* @param mixed $values
* @return mixed
*/
protected function decodeValues($values)
{
if (is_array($values)) {
foreach ($values as &$value) {
$value = $this->decodeValues($value);
}
return $values;
}
return html_entity_decode($values, ENT_COMPAT, 'UTF-8');
}
/**
* Encode HTML entities
*
* @param mixed $values
* @return mixed
*/
protected function encodeValues($values)
{
if (is_array($values)) {
foreach ($values as &$value) {
$value = $this->encodeValues($value);
}
} else {
$values = htmlentities($values, ENT_COMPAT, 'UTF-8');
}
return $values;
}
/**
* Returns a configuration value or section by name.
*
@ -385,85 +305,32 @@ class Config extends Singleton
*/
public function &__get($name)
{
if (!$this->initialized) {
$this->init();
// must be called here, not in init(), since setTestEnvironment() calls init(). (this avoids
// infinite recursion)
Piwik::postTestEvent('Config.createConfigSingleton',
array($this, &$this->configCache, &$this->configLocal));
}
// check cache for merged section
if (isset($this->configCache[$name])) {
$tmp =& $this->configCache[$name];
return $tmp;
}
$section = $this->getFromGlobalConfig($name);
$sectionCommon = $this->getFromCommonConfig($name);
if(empty($section) && !empty($sectionCommon)) {
$section = $sectionCommon;
} elseif(!empty($section) && !empty($sectionCommon)) {
$section = $this->array_merge_recursive_distinct($section, $sectionCommon);
}
if (isset($this->configLocal[$name])) {
// local settings override the global defaults
$section = $section
? array_merge($section, $this->configLocal[$name])
: $this->configLocal[$name];
}
if ($section === null && $name = 'superuser') {
$user = $this->getConfigSuperUserForBackwardCompatibility();
return $user;
} else if ($section === null) {
throw new Exception("Error while trying to read a specific config file entry <strong>'$name'</strong> from your configuration files.</b>If you just completed a Piwik upgrade, please check that the file config/global.ini.php was overwritten by the latest Piwik version.");
}
// cache merged section for later
$this->configCache[$name] = $this->decodeValues($section);
$tmp =& $this->configCache[$name];
return $tmp;
$section =& $this->settings->getIniFileChain()->get($name);
return $section;
}
/**
* @deprecated since version 2.0.4
* @api
*/
public function getConfigSuperUserForBackwardCompatibility()
{
try {
$db = Db::get();
$user = $db->fetchRow("SELECT login, email, password
FROM " . Common::prefixTable("user") . "
WHERE superuser_access = 1
ORDER BY date_registered ASC LIMIT 1");
if (!empty($user)) {
$user['bridge'] = 1;
return $user;
}
} catch (Exception $e) {}
return array();
}
public function getFromGlobalConfig($name)
{
if (isset($this->configGlobal[$name])) {
return $this->configGlobal[$name];
}
return null;
return $this->settings->getIniFileChain()->getFrom($this->getGlobalPath(), $name);
}
/**
* @api
*/
public function getFromCommonConfig($name)
{
if (isset($this->configCommon[$name])) {
return $this->configCommon[$name];
}
return null;
return $this->settings->getIniFileChain()->getFrom($this->getCommonPath(), $name);
}
/**
* @api
*/
public function getFromLocalConfig($name)
{
return $this->settings->getIniFileChain()->getFrom($this->getLocalPath(), $name);
}
/**
@ -475,154 +342,24 @@ class Config extends Singleton
*/
public function __set($name, $value)
{
$this->configCache[$name] = $value;
}
/**
* Comparison function
*
* @param mixed $elem1
* @param mixed $elem2
* @return int;
*/
public static function compareElements($elem1, $elem2)
{
if (is_array($elem1)) {
if (is_array($elem2)) {
return strcmp(serialize($elem1), serialize($elem2));
}
return 1;
}
if (is_array($elem2)) {
return -1;
}
if ((string)$elem1 === (string)$elem2) {
return 0;
}
return ((string)$elem1 > (string)$elem2) ? 1 : -1;
}
/**
* Compare arrays and return difference, such that:
*
* $modified = array_merge($original, $difference);
*
* @param array $original original array
* @param array $modified modified array
* @return array differences between original and modified
*/
public function array_unmerge($original, $modified)
{
// return key/value pairs for keys in $modified but not in $original
// return key/value pairs for keys in both $modified and $original, but values differ
// ignore keys that are in $original but not in $modified
return array_udiff_assoc($modified, $original, array(__CLASS__, 'compareElements'));
$this->settings->getIniFileChain()->set($name, $value);
}
/**
* Dump config
*
* @param array $configLocal
* @param array $configGlobal
* @param array $configCommon
* @param array $configCache
* @return string
* @return string|null
* @throws \Exception
*/
public function dumpConfig($configLocal, $configGlobal, $configCommon, $configCache)
public function dumpConfig()
{
$dirty = false;
$chain = $this->settings->getIniFileChain();
$output = "; <?php exit; ?> DO NOT REMOVE THIS LINE\n";
$output .= "; file automatically generated or modified by Piwik; you can manually override the default values in global.ini.php by redefining them in this file.\n";
if (!$configCache) {
return false;
}
// If there is a common.config.ini.php, this will ensure config.ini.php does not duplicate its values
if(!empty($configCommon)) {
$configGlobal = $this->array_merge_recursive_distinct($configGlobal, $configCommon);
}
if ($configLocal) {
foreach ($configLocal as $name => $section) {
if (!isset($configCache[$name])) {
$configCache[$name] = $this->decodeValues($section);
}
}
}
$sectionNames = array_unique(array_merge(array_keys($configGlobal), array_keys($configCache)));
foreach ($sectionNames as $section) {
if (!isset($configCache[$section])) {
continue;
}
// Only merge if the section exists in global.ini.php (in case a section only lives in config.ini.php)
// get local and cached config
$local = isset($configLocal[$section]) ? $configLocal[$section] : array();
$config = $configCache[$section];
// remove default values from both (they should not get written to local)
if (isset($configGlobal[$section])) {
$config = $this->array_unmerge($configGlobal[$section], $configCache[$section]);
$local = $this->array_unmerge($configGlobal[$section], $local);
}
// if either local/config have non-default values and the other doesn't,
// OR both have values, but different values, we must write to config.ini.php
if (empty($local) xor empty($config)
|| (!empty($local)
&& !empty($config)
&& self::compareElements($config, $configLocal[$section]))
) {
$dirty = true;
}
// no point in writing empty sections, so skip if the cached section is empty
if (empty($config)) {
continue;
}
$output .= "[$section]\n";
foreach ($config as $name => $value) {
$value = $this->encodeValues($value);
if (is_numeric($name)) {
$name = $section;
$value = array($value);
}
if (is_array($value)) {
foreach ($value as $currentValue) {
$output .= $name . "[] = \"$currentValue\"\n";
}
} else {
if (!is_numeric($value)) {
$value = "\"$value\"";
}
$output .= $name . ' = ' . $value . "\n";
}
}
$output .= "\n";
}
if ($dirty) {
return $output;
}
return false;
$header = "; <?php exit; ?> DO NOT REMOVE THIS LINE\n";
$header .= "; file automatically generated or modified by Piwik; you can manually override the default values in global.ini.php by redefining them in this file.\n";
return $chain->dumpChanges($header);
}
/**
* Write user configuration file
*
@ -635,22 +372,24 @@ class Config extends Singleton
*
* @throws \Exception if config file not writable
*/
protected function writeConfig($configLocal, $configGlobal, $configCommon, $configCache, $pathLocal, $clear = true)
protected function writeConfig($clear = true)
{
if ($this->isTest) {
if ($this->doNotWriteConfigInTests) {
return;
}
$output = $this->dumpConfig($configLocal, $configGlobal, $configCommon, $configCache);
if ($output !== false) {
$success = @file_put_contents($pathLocal, $output);
if (!$success) {
$output = $this->dumpConfig();
if ($output !== null
&& $output !== false
) {
$success = @file_put_contents($this->getLocalPath(), $output);
if ($success === false) {
throw $this->getConfigNotWritableException();
}
}
if ($clear) {
$this->clear();
$this->reload();
}
}
@ -662,7 +401,7 @@ class Config extends Singleton
*/
public function forceSave()
{
$this->writeConfig($this->configLocal, $this->configGlobal, $this->configCommon, $this->configCache, $this->pathLocal);
$this->writeConfig();
}
/**
@ -670,45 +409,7 @@ class Config extends Singleton
*/
public function getConfigNotWritableException()
{
$path = "config/" . basename($this->pathLocal);
$path = "config/" . basename($this->getLocalPath());
return new Exception(Piwik::translate('General_ConfigFileIsNotWritable', array("(" . $path . ")", "")));
}
/**
* array_merge_recursive does indeed merge arrays, but it converts values with duplicate
* keys to arrays rather than overwriting the value in the first array with the duplicate
* value in the second array, as array_merge does. I.e., with array_merge_recursive,
* this happens (documented behavior):
*
* array_merge_recursive(array('key' => 'org value'), array('key' => 'new value'));
* => array('key' => array('org value', 'new value'));
*
* array_merge_recursive_distinct does not change the datatypes of the values in the arrays.
* Matching keys' values in the second array overwrite those in the first array, as is the
* case with array_merge, i.e.:
*
* array_merge_recursive_distinct(array('key' => 'org value'), array('key' => 'new value'));
* => array('key' => array('new value'));
*
* Parameters are passed by reference, though only for performance reasons. They're not
* altered by this function.
*
* @param array $array1
* @param array $array2
* @return array
* @author Daniel <daniel (at) danielsmedegaardbuus (dot) dk>
* @author Gabriel Sobrinho <gabriel (dot) sobrinho (at) gmail (dot) com>
*/
function array_merge_recursive_distinct ( array &$array1, array &$array2 )
{
$merged = $array1;
foreach ( $array2 as $key => &$value ) {
if ( is_array ( $value ) && isset ( $merged [$key] ) && is_array ( $merged [$key] ) ) {
$merged [$key] = $this->array_merge_recursive_distinct ( $merged [$key], $value );
} else {
$merged [$key] = $value;
}
}
return $merged;
}
}

16
www/analytics/core/Config/ConfigNotFoundException.php Normal file
View file

@ -0,0 +1,16 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Config;
/**
* Exception thrown when the config file doesn't exist.
*/
class ConfigNotFoundException extends \Exception
{
}

474
www/analytics/core/Config/IniFileChain.php Normal file
View file

@ -0,0 +1,474 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Config;
use Piwik\Common;
use Piwik\Ini\IniReader;
use Piwik\Ini\IniReadingException;
use Piwik\Ini\IniWriter;
/**
* Manages a list of INI files where the settings in each INI file merge with or override the
* settings in the previous INI file.
*
* The IniFileChain class manages two types of INI files: multiple default setting files and one
* user settings file.
*
* The default setting files (for example, global.ini.php & common.ini.php) hold the default setting values.
* The settings in these files are merged recursively, however, array settings in one file will still
* overwrite settings in the previous file.
*
* Default settings files cannot be modified through the IniFileChain class.
*
* The user settings file (for example, config.ini.php) holds the actual setting values. Settings in the
* user settings files overwrite other settings. So array settings will not merge w/ previous values.
*
* HTML characters and dollar signs are stored as encoded HTML entities in INI files. This prevents
* several `parse_ini_file` issues, including one where parse_ini_file tries to insert a variable
* into a setting value if a string like `"$varname" is present.
*/
class IniFileChain
{
/**
* Maps INI file names with their parsed contents. The order of the files signifies the order
* in the chain. Files with lower index are overwritten/merged with files w/ a higher index.
*
* @var array
*/
protected $settingsChain = array();
/**
* The merged INI settings.
*
* @var array
*/
protected $mergedSettings = array();
/**
* Constructor.
*
* @param string[] $defaultSettingsFiles The list of paths to INI files w/ the default setting values.
* @param string|null $userSettingsFile The path to the user settings file.
*/
public function __construct(array $defaultSettingsFiles = array(), $userSettingsFile = null)
{
$this->reload($defaultSettingsFiles, $userSettingsFile);
}
/**
* Return setting section by reference.
*
* @param string $name
* @return mixed
*/
public function &get($name)
{
if (!isset($this->mergedSettings[$name])) {
$this->mergedSettings[$name] = array();
}
$result =& $this->mergedSettings[$name];
return $result;
}
/**
* Return setting section from a specific file, rather than the current merged settings.
*
* @param string $file The path of the file. Should be the path used in construction or reload().
* @param string $name The name of the section to access.
*/
public function getFrom($file, $name)
{
return @$this->settingsChain[$file][$name];
}
/**
* Sets a setting value.
*
* @param string $name
* @param mixed $value
*/
public function set($name, $value)
{
$this->mergedSettings[$name] = $value;
}
/**
* Returns all settings. Changes made to the array result will be reflected in the
* IniFileChain instance.
*
* @return array
*/
public function &getAll()
{
return $this->mergedSettings;
}
/**
* Dumps the current in-memory setting values to a string in INI format and returns it.
*
* @param string $header The header of the output INI file.
* @return string The dumped INI contents.
*/
public function dump($header = '')
{
return $this->dumpSettings($this->mergedSettings, $header);
}
/**
* Writes the difference of the in-memory setting values and the on-disk user settings file setting
* values to a string in INI format, and returns it.
*
* If a config section is identical to the default settings section (as computed by merging
* all default setting files), it is not written to the user settings file.
*
* @param string $header The header of the INI output.
* @return string The dumped INI contents.
*/
public function dumpChanges($header = '')
{
$userSettingsFile = $this->getUserSettingsFile();
$defaultSettings = $this->getMergedDefaultSettings();
$existingMutableSettings = $this->settingsChain[$userSettingsFile];
$dirty = false;
$configToWrite = array();
foreach ($this->mergedSettings as $sectionName => $changedSection) {
if(isset($existingMutableSettings[$sectionName])){
$existingMutableSection = $existingMutableSettings[$sectionName];
} else{
$existingMutableSection = array();
}
// remove default values from both (they should not get written to local)
if (isset($defaultSettings[$sectionName])) {
$changedSection = $this->arrayUnmerge($defaultSettings[$sectionName], $changedSection);
$existingMutableSection = $this->arrayUnmerge($defaultSettings[$sectionName], $existingMutableSection);
}
// if either local/config have non-default values and the other doesn't,
// OR both have values, but different values, we must write to config.ini.php
if (empty($changedSection) xor empty($existingMutableSection)
|| (!empty($changedSection)
&& !empty($existingMutableSection)
&& self::compareElements($changedSection, $existingMutableSection))
) {
$dirty = true;
}
$configToWrite[$sectionName] = $changedSection;
}
if ($dirty) {
// sort config sections by how early they appear in the file chain
$self = $this;
uksort($configToWrite, function ($sectionNameLhs, $sectionNameRhs) use ($self) {
$lhsIndex = $self->findIndexOfFirstFileWithSection($sectionNameLhs);
$rhsIndex = $self->findIndexOfFirstFileWithSection($sectionNameRhs);
if ($lhsIndex == $rhsIndex) {
$lhsIndexInFile = $self->getIndexOfSectionInFile($lhsIndex, $sectionNameLhs);
$rhsIndexInFile = $self->getIndexOfSectionInFile($rhsIndex, $sectionNameRhs);
if ($lhsIndexInFile == $rhsIndexInFile) {
return 0;
} elseif ($lhsIndexInFile < $rhsIndexInFile) {
return -1;
} else {
return 1;
}
} elseif ($lhsIndex < $rhsIndex) {
return -1;
} else {
return 1;
}
});
return $this->dumpSettings($configToWrite, $header);
} else {
return null;
}
}
/**
* Reloads settings from disk.
*/
public function reload($defaultSettingsFiles = array(), $userSettingsFile = null)
{
if (!empty($defaultSettingsFiles)
|| !empty($userSettingsFile)
) {
$this->resetSettingsChain($defaultSettingsFiles, $userSettingsFile);
}
$reader = new IniReader();
foreach ($this->settingsChain as $file => $ignore) {
if (is_readable($file)) {
try {
$contents = $reader->readFile($file);
$this->settingsChain[$file] = $this->decodeValues($contents);
} catch (IniReadingException $ex) {
throw new IniReadingException('Unable to read INI file {' . $file . '}: ' . $ex->getMessage() . "\n Your host may have disabled parse_ini_file().");
}
$this->decodeValues($this->settingsChain[$file]);
}
}
$merged = $this->mergeFileSettings();
// remove reference to $this->settingsChain... otherwise dump() or compareElements() will never notice a difference
// on PHP 7+ as they would be always equal
$this->mergedSettings = $this->copy($merged);
}
private function copy($merged)
{
$copy = array();
foreach ($merged as $index => $value) {
if (is_array($value)) {
$copy[$index] = $this->copy($value);
} else {
$copy[$index] = $value;
}
}
return $copy;
}
private function resetSettingsChain($defaultSettingsFiles, $userSettingsFile)
{
$this->settingsChain = array();
if (!empty($defaultSettingsFiles)) {
foreach ($defaultSettingsFiles as $file) {
$this->settingsChain[$file] = null;
}
}
if (!empty($userSettingsFile)) {
$this->settingsChain[$userSettingsFile] = null;
}
}
protected function mergeFileSettings()
{
$mergedSettings = $this->getMergedDefaultSettings();
$userSettings = end($this->settingsChain) ?: array();
foreach ($userSettings as $sectionName => $section) {
if (!isset($mergedSettings[$sectionName])) {
$mergedSettings[$sectionName] = $section;
} else {
// the last user settings file completely overwrites INI sections. the other files in the chain
// can add to array options
$mergedSettings[$sectionName] = array_merge($mergedSettings[$sectionName], $section);
}
}
return $mergedSettings;
}
protected function getMergedDefaultSettings()
{
$userSettingsFile = $this->getUserSettingsFile();
$mergedSettings = array();
foreach ($this->settingsChain as $file => $settings) {
if ($file == $userSettingsFile
|| empty($settings)
) {
continue;
}
foreach ($settings as $sectionName => $section) {
if (!isset($mergedSettings[$sectionName])) {
$mergedSettings[$sectionName] = $section;
} else {
$mergedSettings[$sectionName] = $this->array_merge_recursive_distinct($mergedSettings[$sectionName], $section);
}
}
}
return $mergedSettings;
}
protected function getUserSettingsFile()
{
// the user settings file is the last key in $settingsChain
end($this->settingsChain);
return key($this->settingsChain);
}
/**
* Comparison function
*
* @param mixed $elem1
* @param mixed $elem2
* @return int;
*/
public static function compareElements($elem1, $elem2)
{
if (is_array($elem1)) {
if (is_array($elem2)) {
return strcmp(serialize($elem1), serialize($elem2));
}
return 1;
}
if (is_array($elem2)) {
return -1;
}
if ((string)$elem1 === (string)$elem2) {
return 0;
}
return ((string)$elem1 > (string)$elem2) ? 1 : -1;
}
/**
* Compare arrays and return difference, such that:
*
* $modified = array_merge($original, $difference);
*
* @param array $original original array
* @param array $modified modified array
* @return array differences between original and modified
*/
public function arrayUnmerge($original, $modified)
{
// return key/value pairs for keys in $modified but not in $original
// return key/value pairs for keys in both $modified and $original, but values differ
// ignore keys that are in $original but not in $modified
return array_udiff_assoc($modified, $original, array(__CLASS__, 'compareElements'));
}
/**
* array_merge_recursive does indeed merge arrays, but it converts values with duplicate
* keys to arrays rather than overwriting the value in the first array with the duplicate
* value in the second array, as array_merge does. I.e., with array_merge_recursive,
* this happens (documented behavior):
*
* array_merge_recursive(array('key' => 'org value'), array('key' => 'new value'));
* => array('key' => array('org value', 'new value'));
*
* array_merge_recursive_distinct does not change the datatypes of the values in the arrays.
* Matching keys' values in the second array overwrite those in the first array, as is the
* case with array_merge, i.e.:
*
* array_merge_recursive_distinct(array('key' => 'org value'), array('key' => 'new value'));
* => array('key' => array('new value'));
*
* Parameters are passed by reference, though only for performance reasons. They're not
* altered by this function.
*
* @param array $array1
* @param array $array2
* @return array
* @author Daniel <daniel (at) danielsmedegaardbuus (dot) dk>
* @author Gabriel Sobrinho <gabriel (dot) sobrinho (at) gmail (dot) com>
*/
private function array_merge_recursive_distinct(array &$array1, array &$array2)
{
$merged = $array1;
foreach ($array2 as $key => &$value) {
if (is_array($value) && isset($merged [$key]) && is_array($merged [$key])) {
$merged [$key] = $this->array_merge_recursive_distinct($merged [$key], $value);
} else {
$merged [$key] = $value;
}
}
return $merged;
}
/**
* public for use in closure.
*/
public function findIndexOfFirstFileWithSection($sectionName)
{
$count = 0;
foreach ($this->settingsChain as $file => $settings) {
if (isset($settings[$sectionName])) {
break;
}
++$count;
}
return $count;
}
/**
* public for use in closure.
*/
public function getIndexOfSectionInFile($fileIndex, $sectionName)
{
reset($this->settingsChain);
for ($i = 0; $i != $fileIndex; ++$i) {
next($this->settingsChain);
}
$settingsData = current($this->settingsChain);
if (empty($settingsData)) {
return -1;
}
$settingsDataSectionNames = array_keys($settingsData);
return array_search($sectionName, $settingsDataSectionNames);
}
/**
* Encode HTML entities
*
* @param mixed $values
* @return mixed
*/
protected function encodeValues(&$values)
{
if (is_array($values)) {
foreach ($values as &$value) {
$value = $this->encodeValues($value);
}
} elseif (is_float($values)) {
$values = Common::forceDotAsSeparatorForDecimalPoint($values);
} elseif (is_string($values)) {
$values = htmlentities($values, ENT_COMPAT, 'UTF-8');
$values = str_replace('$', '&#36;', $values);
}
return $values;
}
/**
* Decode HTML entities
*
* @param mixed $values
* @return mixed
*/
protected function decodeValues(&$values)
{
if (is_array($values)) {
foreach ($values as &$value) {
$value = $this->decodeValues($value);
}
return $values;
} elseif (is_string($values)) {
return html_entity_decode($values, ENT_COMPAT, 'UTF-8');
}
return $values;
}
private function dumpSettings($values, $header)
{
$values = $this->encodeValues($values);
$writer = new IniWriter();
return $writer->writeToString($values, $header);
}
}

162
www/analytics/core/Console.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,19 +8,32 @@
*/
namespace Piwik;
use Symfony\Component\Console\Application;
use Symfony\Component\Console\Input\ArgvInput;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Piwik\Application\Environment;
use Piwik\Config\ConfigNotFoundException;
use Piwik\Container\StaticContainer;
use Piwik\Plugin\Manager as PluginManager;
use Symfony\Bridge\Monolog\Handler\ConsoleHandler;
use Symfony\Component\Console\Application;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class Console extends Application
{
public function __construct()
/**
* @var Environment
*/
private $environment;
public function __construct(Environment $environment = null)
{
$this->setServerArgsIfPhpCgi();
parent::__construct();
$this->environment = $environment;
$option = new InputOption('piwik-domain',
null,
InputOption::VALUE_OPTIONAL,
@ -28,42 +41,60 @@ class Console extends Application
);
$this->getDefinition()->addOption($option);
}
/**
* @deprecated
*/
public function init()
{
// TODO: remove
$option = new InputOption('xhprof',
null,
InputOption::VALUE_NONE,
'Enable profiling with XHProf'
);
$this->getDefinition()->addOption($option);
}
public function doRun(InputInterface $input, OutputInterface $output)
{
$this->initPiwikHost($input);
$this->initConfig($output);
try {
self::initPlugins();
} catch(\Exception $e) {
// Piwik not installed yet, no config file?
if ($input->hasParameterOption('--xhprof')) {
Profiler::setupProfilerXHProf(true, true);
}
Translate::reloadLanguage('en');
$this->initPiwikHost($input);
$this->initEnvironment($output);
$this->initLoggerOutput($output);
try {
self::initPlugins();
} catch (ConfigNotFoundException $e) {
// Piwik not installed yet, no config file?
Log::warning($e->getMessage());
}
$commands = $this->getAvailableCommands();
foreach ($commands as $command) {
if (!class_exists($command)) {
Log::warning(sprintf('Cannot add command %s, class does not exist', $command));
} elseif (!is_subclass_of($command, 'Piwik\Plugin\ConsoleCommand')) {
Log::warning(sprintf('Cannot add command %s, class does not extend Piwik\Plugin\ConsoleCommand', $command));
} else {
$this->add(new $command);
}
$this->addCommandIfExists($command);
}
return parent::doRun($input, $output);
$self = $this;
return Access::doAsSuperUser(function () use ($input, $output, $self) {
return call_user_func(array($self, 'Symfony\Component\Console\Application::doRun'), $input, $output);
});
}
private function addCommandIfExists($command)
{
if (!class_exists($command)) {
Log::warning(sprintf('Cannot add command %s, class does not exist', $command));
} elseif (!is_subclass_of($command, 'Piwik\Plugin\ConsoleCommand')) {
Log::warning(sprintf('Cannot add command %s, class does not extend Piwik\Plugin\ConsoleCommand', $command));
} else {
/** @var Command $commandInstance */
$commandInstance = new $command;
// do not add the command if it already exists; this way we can add the command ourselves in tests
if (!$this->has($commandInstance->getName())) {
$this->add($commandInstance);
}
}
}
/**
@ -74,11 +105,9 @@ class Console extends Application
private function getAvailableCommands()
{
$commands = $this->getDefaultPiwikCommands();
$detected = PluginManager::getInstance()->findMultipleComponents('Commands', 'Piwik\\Plugin\\ConsoleCommand');
$pluginNames = PluginManager::getInstance()->getLoadedPluginsName();
foreach ($pluginNames as $pluginName) {
$commands = array_merge($commands, $this->findCommandsInPlugin($pluginName));
}
$commands = array_merge($commands, $detected);
/**
* Triggered to filter / restrict console commands. Plugins that want to restrict commands
@ -103,52 +132,76 @@ class Console extends Application
return $commands;
}
private function findCommandsInPlugin($pluginName)
private function setServerArgsIfPhpCgi()
{
$commands = array();
if (Common::isPhpCgiType()) {
$_SERVER['argv'] = array();
foreach ($_GET as $name => $value) {
$argument = $name;
if (!empty($value)) {
$argument .= '=' . $value;
}
$files = Filesystem::globr(PIWIK_INCLUDE_PATH . '/plugins/' . $pluginName .'/Commands', '*.php');
foreach ($files as $file) {
$klassName = sprintf('Piwik\\Plugins\\%s\\Commands\\%s', $pluginName, basename($file, '.php'));
if (!class_exists($klassName) || !is_subclass_of($klassName, 'Piwik\\Plugin\\ConsoleCommand')) {
continue;
$_SERVER['argv'][] = $argument;
}
$klass = new \ReflectionClass($klassName);
if ($klass->isAbstract()) {
continue;
if (!defined('STDIN')) {
define('STDIN', fopen('php://stdin', 'r'));
}
$commands[] = $klassName;
}
}
return $commands;
public static function isSupported()
{
return Common::isPhpCliMode() && !Common::isPhpCgiType();
}
protected function initPiwikHost(InputInterface $input)
{
$piwikHostname = $input->getParameterOption('--piwik-domain');
if (empty($piwikHostname)) {
$piwikHostname = $input->getParameterOption('--url');
}
$piwikHostname = UrlHelper::getHostFromUrl($piwikHostname);
Url::setHost($piwikHostname);
}
protected function initConfig(OutputInterface $output)
protected function initEnvironment(OutputInterface $output)
{
$config = Config::getInstance();
try {
$config->checkLocalConfigFound();
if ($this->environment === null) {
$this->environment = new Environment('cli');
$this->environment->init();
}
$config = Config::getInstance();
return $config;
} catch (\Exception $e) {
$output->writeln($e->getMessage() . "\n");
}
}
/**
* Register the console output into the logger.
*
* Ideally, this should be done automatically with events:
* @see http://symfony.com/fr/doc/current/components/console/events.html
* @see Symfony\Bridge\Monolog\Handler\ConsoleHandler::onCommand()
* But it would require to install Symfony's Event Dispatcher.
*/
private function initLoggerOutput(OutputInterface $output)
{
/** @var ConsoleHandler $consoleLogHandler */
$consoleLogHandler = StaticContainer::get('Symfony\Bridge\Monolog\Handler\ConsoleHandler');
$consoleLogHandler->setOutput($output);
}
public static function initPlugins()
{
Plugin\Manager::getInstance()->loadActivatedPlugins();
Plugin\Manager::getInstance()->loadPluginTranslations();
}
private function getDefaultPiwikCommands()
@ -156,11 +209,12 @@ class Console extends Application
$commands = array(
'Piwik\CliMulti\RequestCommand'
);
if (class_exists('Piwik\Plugins\EnterpriseAdmin\EnterpriseAdmin')) {
$extra = new \Piwik\Plugins\EnterpriseAdmin\EnterpriseAdmin();
$extra->addConsoleCommands($commands);
}
return $commands;
}
}

18
www/analytics/core/Container/ContainerDoesNotExistException.php Normal file
View file

@ -0,0 +1,18 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Container;
use RuntimeException;
/**
* Thrown if the root container has not been created and set in StaticContainer.
*/
class ContainerDoesNotExistException extends RuntimeException
{
}

151
www/analytics/core/Container/ContainerFactory.php Normal file
View file

@ -0,0 +1,151 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Container;
use DI\Container;
use DI\ContainerBuilder;
use Doctrine\Common\Cache\ArrayCache;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
use Piwik\Plugin\Manager;
/**
* Creates a configured DI container.
*/
class ContainerFactory
{
/**
* @var PluginList
*/
private $pluginList;
/**
* @var GlobalSettingsProvider
*/
private $settings;
/**
* Optional environment configs to load.
*
* @var string[]
*/
private $environments;
/**
* @var array[]
*/
private $definitions;
/**
* @param PluginList $pluginList
* @param GlobalSettingsProvider $settings
* @param string[] $environment Optional environment configs to load.
* @param array[] $definitions
*/
public function __construct(PluginList $pluginList, GlobalSettingsProvider $settings, array $environments = array(), array $definitions = array())
{
$this->pluginList = $pluginList;
$this->settings = $settings;
$this->environments = $environments;
$this->definitions = $definitions;
}
/**
* @link http://php-di.org/doc/container-configuration.html
* @throws \Exception
* @return Container
*/
public function create()
{
$builder = new ContainerBuilder();
$builder->useAnnotations(false);
$builder->setDefinitionCache(new ArrayCache());
// INI config
$builder->addDefinitions(new IniConfigDefinitionSource($this->settings));
// Global config
$builder->addDefinitions(PIWIK_USER_PATH . '/config/global.php');
// Plugin configs
$this->addPluginConfigs($builder);
// Development config
if ($this->isDevelopmentModeEnabled()) {
$this->addEnvironmentConfig($builder, 'dev');
}
// Environment config
foreach ($this->environments as $environment) {
$this->addEnvironmentConfig($builder, $environment);
}
// User config
if (file_exists(PIWIK_USER_PATH . '/config/config.php')) {
$builder->addDefinitions(PIWIK_USER_PATH . '/config/config.php');
}
if (!empty($this->definitions)) {
foreach ($this->definitions as $definitionArray) {
$builder->addDefinitions($definitionArray);
}
}
$container = $builder->build();
$container->set('Piwik\Application\Kernel\PluginList', $this->pluginList);
$container->set('Piwik\Application\Kernel\GlobalSettingsProvider', $this->settings);
return $container;
}
private function addEnvironmentConfig(ContainerBuilder $builder, $environment)
{
if (!$environment) {
return;
}
$file = sprintf('%s/config/environment/%s.php', PIWIK_USER_PATH, $environment);
if (file_exists($file)) {
$builder->addDefinitions($file);
}
// add plugin environment configs
$plugins = $this->pluginList->getActivatedPlugins();
foreach ($plugins as $plugin) {
$baseDir = Manager::getPluginsDirectory() . $plugin;
$environmentFile = $baseDir . '/config/' . $environment . '.php';
if (file_exists($environmentFile)) {
$builder->addDefinitions($environmentFile);
}
}
}
private function addPluginConfigs(ContainerBuilder $builder)
{
$plugins = $this->pluginList->getActivatedPlugins();
foreach ($plugins as $plugin) {
$baseDir = Manager::getPluginsDirectory() . $plugin;
$file = $baseDir . '/config/config.php';
if (file_exists($file)) {
$builder->addDefinitions($file);
}
}
}
private function isDevelopmentModeEnabled()
{
$section = $this->settings->getSection('Development');
return (bool) @$section['enabled']; // TODO: code redundancy w/ Development. hopefully ok for now.
}
}

95
www/analytics/core/Container/IniConfigDefinitionSource.php Normal file
View file

@ -0,0 +1,95 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Container;
use DI\Definition\Exception\DefinitionException;
use DI\Definition\Source\DefinitionSource;
use DI\Definition\ValueDefinition;
use Piwik\Application\Kernel\GlobalSettingsProvider;
/**
* Expose the INI config into PHP-DI.
*
* The INI config can be used by prefixing `ini.` before the setting we want to get:
*
* $maintenanceMode = $container->get('ini.General.maintenance_mode');
*/
class IniConfigDefinitionSource implements DefinitionSource
{
/**
* @var GlobalSettingsProvider
*/
private $config;
/**
* @var string
*/
private $prefix;
/**
* @param GlobalSettingsProvider $config
* @param string $prefix Prefix for the container entries.
*/
public function __construct(GlobalSettingsProvider $config, $prefix = 'ini.')
{
$this->config = $config;
$this->prefix = $prefix;
}
/**
* {@inheritdoc}
*/
public function getDefinition($name)
{
if (strpos($name, $this->prefix) !== 0) {
return null;
}
list($sectionName, $configKey) = $this->parseEntryName($name);
$section = $this->getSection($sectionName);
if ($configKey === null) {
return new ValueDefinition($name, $section);
}
if (! array_key_exists($configKey, $section)) {
return null;
}
return new ValueDefinition($name, $section[$configKey]);
}
private function parseEntryName($name)
{
$parts = explode('.', $name, 3);
array_shift($parts);
if (! isset($parts[1])) {
$parts[1] = null;
}
return $parts;
}
private function getSection($sectionName)
{
$section = $this->config->getSection($sectionName);
if (!is_array($section)) {
throw new DefinitionException(sprintf(
'IniFileChain did not return an array for the config section %s',
$section
));
}
return $section;
}
}

87
www/analytics/core/Container/StaticContainer.php Normal file
View file

@ -0,0 +1,87 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Container;
use DI\Container;
/**
* This class provides a static access to the container.
*
* @deprecated This class is introduced only to keep BC with the current static architecture. It will be removed in 3.0.
* - it is global state (that class makes the container a global variable)
* - using the container directly is the "service locator" anti-pattern (which is not dependency injection)
*/
class StaticContainer
{
/**
* @var Container[]
*/
private static $containerStack = array();
/**
* Definitions to register in the container.
*
* @var array[]
*/
private static $definitions = array();
/**
* @return Container
*/
public static function getContainer()
{
if (empty(self::$containerStack)) {
throw new ContainerDoesNotExistException("The root container has not been created yet.");
}
return end(self::$containerStack);
}
public static function clearContainer()
{
self::pop();
}
/**
* Only use this in tests.
*
* @param Container $container
*/
public static function push(Container $container)
{
self::$containerStack[] = $container;
}
public static function pop()
{
array_pop(self::$containerStack);
}
public static function addDefinitions(array $definitions)
{
self::$definitions[] = $definitions;
}
/**
* Proxy to Container::get()
*
* @param string $name Container entry name.
* @return mixed
* @throws \DI\NotFoundException
*/
public static function get($name)
{
return self::getContainer()->get($name);
}
public static function getDefinitions()
{
return self::$definitions;
}
}

34
www/analytics/core/Cookie.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -143,7 +143,9 @@ class Cookie
// Remove port information.
$Port = strpos($Domain, ':');
if ($Port !== false) $Domain = substr($Domain, 0, $Port);
if ($Port !== false) {
$Domain = substr($Domain, 0, $Port);
}
}
$header = 'Set-Cookie: ' . rawurlencode($Name) . '=' . rawurlencode($Value)
@ -200,12 +202,14 @@ class Cookie
private function extractSignedContent($content)
{
$signature = substr($content, -40);
if (substr($content, -43, 3) == self::VALUE_SEPARATOR . '_=' &&
$signature == sha1(substr($content, 0, -40) . SettingsPiwik::getSalt())
) {
// strip trailing: VALUE_SEPARATOR '_=' signature"
return substr($content, 0, -43);
}
return false;
}
@ -218,6 +222,7 @@ class Cookie
protected function loadContentFromCookie()
{
$cookieStr = $this->extractSignedContent($_COOKIE[$this->name]);
if ($cookieStr === false) {
return;
}
@ -255,6 +260,7 @@ class Cookie
protected function generateContentString()
{
$cookieStr = '';
foreach ($this->value as $name => $value) {
if (!is_numeric($value)) {
$value = base64_encode(safe_serialize($value));
@ -335,6 +341,7 @@ class Cookie
$this->value[$name] = $value;
return;
}
$this->value[$this->keyStore][$name] = $value;
}
@ -347,14 +354,19 @@ class Cookie
public function get($name)
{
$name = self::escapeValue($name);
if ($this->keyStore === false) {
return isset($this->value[$name])
? self::escapeValue($this->value[$name])
: false;
if (false === $this->keyStore) {
if (isset($this->value[$name])) {
return self::escapeValue($this->value[$name]);
}
return false;
}
return isset($this->value[$this->keyStore][$name])
? self::escapeValue($this->value[$this->keyStore][$name])
: false;
if (isset($this->value[$this->keyStore][$name])) {
return self::escapeValue($this->value[$this->keyStore][$name]);
}
return false;
}
/**
@ -364,8 +376,10 @@ class Cookie
*/
public function __toString()
{
$str = 'COOKIE ' . $this->name . ', rows count: ' . count($this->value) . ', cookie size = ' . strlen($this->generateContentString()) . " bytes\n";
$str = 'COOKIE ' . $this->name . ', rows count: ' . count($this->value) . ', cookie size = ' . strlen($this->generateContentString()) . " bytes, ";
$str .= 'path: ' . $this->path. ', expire: ' . $this->expire . "\n";
$str .= var_export($this->value, $return = true);
return $str;
}

1740
www/analytics/core/CronArchive.php
View file

File diff suppressed because it is too large Load diff

5
www/analytics/core/CronArchive/FixedSiteIds.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -8,7 +8,6 @@
*/
namespace Piwik\CronArchive;
use Piwik\CronArchive;
class FixedSiteIds
{
@ -63,6 +62,4 @@ class FixedSiteIds
return null;
}
}

208
www/analytics/core/CronArchive/SegmentArchivingRequestUrlProvider.php Normal file
View file

@ -0,0 +1,208 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CronArchive;
use Piwik\Cache\Cache;
use Piwik\Cache\Transient;
use Piwik\Container\StaticContainer;
use Piwik\Date;
use Piwik\Period\Factory as PeriodFactory;
use Piwik\Period\Range;
use Piwik\Plugins\SegmentEditor\Model;
use Psr\Log\LoggerInterface;
/**
* Provides URLs that initiate archiving during cron archiving for segments.
*
* Handles the `[General] process_new_segments_from` INI option.
*/
class SegmentArchivingRequestUrlProvider
{
const BEGINNING_OF_TIME = 'beginning_of_time';
const CREATION_TIME = 'segment_creation_time';
const LAST_EDIT_TIME = 'segment_last_edit_time';
/**
* @var Model
*/
private $segmentEditorModel;
/**
* @var Cache
*/
private $segmentListCache;
/**
* @var Date
*/
private $now;
private $processNewSegmentsFrom;
/**
* @var LoggerInterface
*/
private $logger;
public function __construct($processNewSegmentsFrom, Model $segmentEditorModel = null, Cache $segmentListCache = null,
Date $now = null, LoggerInterface $logger = null)
{
$this->processNewSegmentsFrom = $processNewSegmentsFrom;
$this->segmentEditorModel = $segmentEditorModel ?: new Model();
$this->segmentListCache = $segmentListCache ?: new Transient();
$this->now = $now ?: Date::factory('now');
$this->logger = $logger ?: StaticContainer::get('Psr\Log\LoggerInterface');
}
public function getUrlParameterDateString($idSite, $period, $date, $segment)
{
$oldestDateToProcessForNewSegment = $this->getOldestDateToProcessForNewSegment($idSite, $segment);
if (empty($oldestDateToProcessForNewSegment)) {
return $date;
}
// if the start date for the archiving request is before the minimum date allowed for processing this segment,
// use the minimum allowed date as the start date
$periodObj = PeriodFactory::build($period, $date);
if ($periodObj->getDateStart()->getTimestamp() < $oldestDateToProcessForNewSegment->getTimestamp()) {
$this->logger->debug("Start date of archiving request period ({start}) is older than configured oldest date to process for the segment.", array(
'start' => $periodObj->getDateStart()
));
$endDate = $periodObj->getDateEnd();
// if the creation time of a segment is older than the end date of the archiving request range, we cannot
// blindly rewrite the date string, since the resulting range would be incorrect. instead we make the
// start date equal to the end date, so less archiving occurs, and no fatal error occurs.
if ($oldestDateToProcessForNewSegment->getTimestamp() > $endDate->getTimestamp()) {
$this->logger->debug("Oldest date to process is greater than end date of archiving request period ({end}), so setting oldest date to end date.", array(
'end' => $endDate
));
$oldestDateToProcessForNewSegment = $endDate;
}
$date = $oldestDateToProcessForNewSegment->toString().','.$endDate;
$this->logger->debug("Archiving request date range changed to {date} w/ period {period}.", array('date' => $date, 'period' => $period));
}
return $date;
}
private function getOldestDateToProcessForNewSegment($idSite, $segment)
{
/**
* @var Date $segmentCreatedTime
* @var Date $segmentLastEditedTime
*/
list($segmentCreatedTime, $segmentLastEditedTime) = $this->getCreatedTimeOfSegment($idSite, $segment);
if ($this->processNewSegmentsFrom == self::CREATION_TIME) {
$this->logger->debug("process_new_segments_from set to segment_creation_time, oldest date to process is {time}", array('time' => $segmentCreatedTime));
return $segmentCreatedTime;
} elseif ($this->processNewSegmentsFrom == self::LAST_EDIT_TIME) {
$this->logger->debug("process_new_segments_from set to segment_last_edit_time, segment last edit time is {time}",
array('time' => $segmentLastEditedTime));
if ($segmentLastEditedTime === null
|| $segmentLastEditedTime->getTimestamp() < $segmentCreatedTime->getTimestamp()
) {
$this->logger->debug("segment last edit time is older than created time, using created time instead");
$segmentLastEditedTime = $segmentCreatedTime;
}
return $segmentLastEditedTime;
} elseif (preg_match("/^last([0-9]+)$/", $this->processNewSegmentsFrom, $matches)) {
$lastN = $matches[1];
list($lastDate, $lastPeriod) = Range::getDateXPeriodsAgo($lastN, $segmentCreatedTime, 'day');
$result = Date::factory($lastDate);
$this->logger->debug("process_new_segments_from set to last{N}, oldest date to process is {time}", array('N' => $lastN, 'time' => $result));
return $result;
} else {
$this->logger->debug("process_new_segments_from set to beginning_of_time or cannot recognize value");
return null;
}
}
private function getCreatedTimeOfSegment($idSite, $segmentDefinition)
{
$segments = $this->getAllSegments();
/** @var Date $latestEditTime */
$latestEditTime = null;
$earliestCreatedTime = $this->now;
foreach ($segments as $segment) {
if (empty($segment['ts_created'])
|| empty($segment['definition'])
|| !isset($segment['enable_only_idsite'])
) {
continue;
}
if ($this->isSegmentForSite($segment, $idSite)
&& $segment['definition'] == $segmentDefinition
) {
// check for an earlier ts_created timestamp
$createdTime = Date::factory($segment['ts_created']);
if ($createdTime->getTimestamp() < $earliestCreatedTime->getTimestamp()) {
$earliestCreatedTime = $createdTime;
}
// if there is no ts_last_edit timestamp, initialize it to ts_created
if (empty($segment['ts_last_edit'])) {
$segment['ts_last_edit'] = $segment['ts_created'];
}
// check for a later ts_last_edit timestamp
$lastEditTime = Date::factory($segment['ts_last_edit']);
if ($latestEditTime === null
|| $latestEditTime->getTimestamp() < $lastEditTime->getTimestamp()
) {
$latestEditTime = $lastEditTime;
}
}
}
$this->logger->debug(
"Earliest created time of segment '{segment}' w/ idSite = {idSite} is found to be {createdTime}. Latest " .
"edit time is found to be {latestEditTime}.",
array(
'segment' => $segmentDefinition,
'idSite' => $idSite,
'createdTime' => $earliestCreatedTime,
'latestEditTime' => $latestEditTime,
)
);
return array($earliestCreatedTime, $latestEditTime);
}
private function getAllSegments()
{
if (!$this->segmentListCache->contains('all')) {
$segments = $this->segmentEditorModel->getAllSegmentsAndIgnoreVisibility();
$this->segmentListCache->save('all', $segments);
}
return $this->segmentListCache->fetch('all');
}
private function isSegmentForSite($segment, $idSite)
{
return $segment['enable_only_idsite'] == 0
|| $segment['enable_only_idsite'] == $idSite;
}
}

24
www/analytics/core/CronArchive/SharedSiteIds.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -18,12 +18,22 @@ use Piwik\Option;
*/
class SharedSiteIds
{
const OPTION_DEFAULT = 'SharedSiteIdsToArchive';
const OPTION_ALL_WEBSITES = 'SharedSiteIdsToArchive_AllWebsites';
/**
* @var string
*/
private $optionName;
private $siteIds = array();
private $currentSiteId;
private $done = false;
public function __construct($websiteIds)
public function __construct($websiteIds, $optionName = self::OPTION_DEFAULT)
{
$this->optionName = $optionName;
if (empty($websiteIds)) {
$websiteIds = array();
}
@ -86,16 +96,16 @@ class SharedSiteIds
public function setSiteIdsToArchive($siteIds)
{
if (!empty($siteIds)) {
Option::set('SharedSiteIdsToArchive', implode(',', $siteIds));
Option::set($this->optionName, implode(',', $siteIds));
} else {
Option::delete('SharedSiteIdsToArchive');
Option::delete($this->optionName);
}
}
public function getAllSiteIdsToArchive()
{
Option::clearCachedOption('SharedSiteIdsToArchive');
$siteIdsToArchive = Option::get('SharedSiteIdsToArchive');
Option::clearCachedOption($this->optionName);
$siteIdsToArchive = Option::get($this->optionName);
if (empty($siteIdsToArchive)) {
return array();
@ -171,6 +181,4 @@ class SharedSiteIds
{
return Process::isSupported();
}
}

40
www/analytics/core/CronArchive/SitesToReprocessDistributedList.php Normal file
View file

@ -0,0 +1,40 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\CronArchive;
use Piwik\Concurrency\DistributedList;
/**
* Distributed list that stores the list of IDs of sites whose archives should be reprocessed.
*
* CronArchive will read this list of sites when archiving is being run, and make sure the sites
* are re-archived.
*
* Any class/API method/command/etc. is allowed to add site IDs to this list.
*/
class SitesToReprocessDistributedList extends DistributedList
{
const OPTION_INVALIDATED_IDSITES_TO_REPROCESS = 'InvalidatedOldReports_WebsiteIds';
public function __construct()
{
parent::__construct(self::OPTION_INVALIDATED_IDSITES_TO_REPROCESS);
}
/**
* @inheritdoc
*/
public function setAll($items)
{
$items = array_unique($items, SORT_REGULAR);
$items = array_values($items);
parent::setAll($items);
}
}

34
www/analytics/core/DataAccess/Actions.php Normal file
View file

@ -0,0 +1,34 @@
<?php
/**
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\DataAccess;
use Piwik\Db;
use Piwik\Common;
/**
* Data Access Object for operations dealing with the log_action table.
*/
class Actions
{
/**
* Removes a list of actions from the log_action table by ID.
*
* @param int[] $idActions
*/
public function delete($idActions)
{
foreach ($idActions as &$id) {
$id = (int)$id;
}
$table = Common::prefixTable('log_action');
$sql = "DELETE FROM $table WHERE idaction IN (" . implode(",", $idActions) . ")";
Db::query($sql);
}
}

281
www/analytics/core/DataAccess/ArchiveSelector.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,18 +9,16 @@
namespace Piwik\DataAccess;
use Exception;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Archive;
use Piwik\Archive\Chunk;
use Piwik\ArchiveProcessor;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Common;
use Piwik\Date;
use Piwik\Db;
use Piwik\Log;
use Piwik\Period;
use Piwik\Period\Range;
use Piwik\Piwik;
use Piwik\Segment;
use Piwik\Site;
/**
* Data Access object used to query archives
@ -42,52 +40,46 @@ class ArchiveSelector
const NB_VISITS_CONVERTED_RECORD_LOOKED_UP = "nb_visits_converted";
static public function getArchiveIdAndVisits(ArchiveProcessor\Parameters $params, $minDatetimeArchiveProcessedUTC)
private static function getModel()
{
$dateStart = $params->getPeriod()->getDateStart();
$bindSQL = array($params->getSite()->getId(),
$dateStart->toString('Y-m-d'),
$params->getPeriod()->getDateEnd()->toString('Y-m-d'),
$params->getPeriod()->getId(),
);
return new Model();
}
$timeStampWhere = '';
public static function getArchiveIdAndVisits(ArchiveProcessor\Parameters $params, $minDatetimeArchiveProcessedUTC)
{
$idSite = $params->getSite()->getId();
$period = $params->getPeriod()->getId();
$dateStart = $params->getPeriod()->getDateStart();
$dateStartIso = $dateStart->toString('Y-m-d');
$dateEndIso = $params->getPeriod()->getDateEnd()->toString('Y-m-d');
$numericTable = ArchiveTableCreator::getNumericTable($dateStart);
$minDatetimeIsoArchiveProcessedUTC = null;
if ($minDatetimeArchiveProcessedUTC) {
$timeStampWhere = " AND ts_archived >= ? ";
$bindSQL[] = Date::factory($minDatetimeArchiveProcessedUTC)->getDatetime();
$minDatetimeIsoArchiveProcessedUTC = Date::factory($minDatetimeArchiveProcessedUTC)->getDatetime();
}
$requestedPlugin = $params->getRequestedPlugin();
$segment = $params->getSegment();
$isSkipAggregationOfSubTables = $params->isSkipAggregationOfSubTables();
$segment = $params->getSegment();
$plugins = array("VisitsSummary", $requestedPlugin);
$sqlWhereArchiveName = self::getNameCondition($plugins, $segment, $isSkipAggregationOfSubTables);
$sqlQuery = " SELECT idarchive, value, name, date1 as startDate
FROM " . ArchiveTableCreator::getNumericTable($dateStart) . "``
WHERE idsite = ?
AND date1 = ?
AND date2 = ?
AND period = ?
AND ( ($sqlWhereArchiveName)
OR name = '" . self::NB_VISITS_RECORD_LOOKED_UP . "'
OR name = '" . self::NB_VISITS_CONVERTED_RECORD_LOOKED_UP . "')
$timeStampWhere
ORDER BY idarchive DESC";
$results = Db::fetchAll($sqlQuery, $bindSQL);
$doneFlags = Rules::getDoneFlags($plugins, $segment);
$doneFlagValues = Rules::getSelectableDoneFlagValues();
$results = self::getModel()->getArchiveIdAndVisits($numericTable, $idSite, $period, $dateStartIso, $dateEndIso, $minDatetimeIsoArchiveProcessedUTC, $doneFlags, $doneFlagValues);
if (empty($results)) {
return false;
}
$idArchive = self::getMostRecentIdArchiveFromResults($segment, $requestedPlugin, $isSkipAggregationOfSubTables, $results);
$idArchiveVisitsSummary = self::getMostRecentIdArchiveFromResults($segment, "VisitsSummary", $isSkipAggregationOfSubTables, $results);
$idArchive = self::getMostRecentIdArchiveFromResults($segment, $requestedPlugin, $results);
$idArchiveVisitsSummary = self::getMostRecentIdArchiveFromResults($segment, "VisitsSummary", $results);
list($visits, $visitsConverted) = self::getVisitsMetricsFromResults($idArchive, $idArchiveVisitsSummary, $results);
if ($visits === false
&& $idArchive === false
) {
if (false === $visits && false === $idArchive) {
return false;
}
@ -98,9 +90,11 @@ class ArchiveSelector
{
$visits = $visitsConverted = false;
$archiveWithVisitsMetricsWasFound = ($idArchiveVisitsSummary !== false);
if ($archiveWithVisitsMetricsWasFound) {
$visits = $visitsConverted = 0;
}
foreach ($results as $result) {
if (in_array($result['idarchive'], array($idArchive, $idArchiveVisitsSummary))) {
$value = (int)$result['value'];
@ -116,13 +110,15 @@ class ArchiveSelector
}
}
}
return array($visits, $visitsConverted);
}
protected static function getMostRecentIdArchiveFromResults(Segment $segment, $requestedPlugin, $isSkipAggregationOfSubTables, $results)
protected static function getMostRecentIdArchiveFromResults(Segment $segment, $requestedPlugin, $results)
{
$idArchive = false;
$namesRequestedPlugin = Rules::getDoneFlags(array($requestedPlugin), $segment, $isSkipAggregationOfSubTables);
$namesRequestedPlugin = Rules::getDoneFlags(array($requestedPlugin), $segment);
foreach ($results as $result) {
if ($idArchive === false
&& in_array($result['name'], $namesRequestedPlugin)
@ -131,6 +127,7 @@ class ArchiveSelector
break;
}
}
return $idArchive;
}
@ -141,21 +138,29 @@ class ArchiveSelector
* @param array $periods
* @param Segment $segment
* @param array $plugins List of plugin names for which data is being requested.
* @param bool $isSkipAggregationOfSubTables Whether we are selecting an archive that may be partial (no sub-tables)
* @return array Archive IDs are grouped by archive name and period range, ie,
* array(
* 'VisitsSummary.done' => array(
* '2010-01-01' => array(1,2,3)
* )
* )
* @throws
*/
static public function getArchiveIds($siteIds, $periods, $segment, $plugins, $isSkipAggregationOfSubTables = false)
public static function getArchiveIds($siteIds, $periods, $segment, $plugins)
{
if (empty($siteIds)) {
throw new \Exception("Website IDs could not be read from the request, ie. idSite=");
}
foreach ($siteIds as $index => $siteId) {
$siteIds[$index] = (int) $siteId;
}
$getArchiveIdsSql = "SELECT idsite, name, date1, date2, MAX(idarchive) as idarchive
FROM %s
WHERE %s
AND " . self::getNameCondition($plugins, $segment, $isSkipAggregationOfSubTables) . "
AND idsite IN (" . implode(',', $siteIds) . ")
WHERE idsite IN (" . implode(',', $siteIds) . ")
AND " . self::getNameCondition($plugins, $segment) . "
AND %s
GROUP BY idsite, date1, date2";
$monthToPeriods = array();
@ -197,14 +202,14 @@ class ArchiveSelector
$sql = sprintf($getArchiveIdsSql, $table, $dateCondition);
$archiveIds = Db::fetchAll($sql, $bind);
// get the archive IDs
foreach (Db::fetchAll($sql, $bind) as $row) {
$archiveName = $row['name'];
foreach ($archiveIds as $row) {
//FIXMEA duplicate with Archive.php
$dateStr = $row['date1'] . "," . $row['date2'];
$dateStr = $row['date1'] . ',' . $row['date2'];
$result[$archiveName][$dateStr][] = $row['idarchive'];
$result[$row['name']][$dateStr][] = $row['idarchive'];
}
}
@ -215,29 +220,52 @@ class ArchiveSelector
* Queries and returns archive data using a set of archive IDs.
*
* @param array $archiveIds The IDs of the archives to get data from.
* @param array $recordNames The names of the data to retrieve (ie, nb_visits, nb_actions, etc.)
* @param array $recordNames The names of the data to retrieve (ie, nb_visits, nb_actions, etc.).
* Note: You CANNOT pass multiple recordnames if $loadAllSubtables=true.
* @param string $archiveDataType The archive data type (either, 'blob' or 'numeric').
* @param bool $loadAllSubtables Whether to pre-load all subtables
* @param int|null|string $idSubtable null if the root blob should be loaded, an integer if a subtable should be
* loaded and 'all' if all subtables should be loaded.
* @throws Exception
* @return array
*/
static public function getArchiveData($archiveIds, $recordNames, $archiveDataType, $loadAllSubtables)
public static function getArchiveData($archiveIds, $recordNames, $archiveDataType, $idSubtable)
{
$chunk = new Chunk();
// create the SQL to select archive data
$inNames = Common::getSqlStringFieldsArray($recordNames);
$loadAllSubtables = $idSubtable == Archive::ID_SUBTABLE_LOAD_ALL_SUBTABLES;
if ($loadAllSubtables) {
$name = reset($recordNames);
// select blobs w/ name like "$name_[0-9]+" w/o using RLIKE
$nameEnd = strlen($name) + 2;
$whereNameIs = "(name = ?
OR (name LIKE ?
AND SUBSTRING(name, $nameEnd, 1) >= '0'
AND SUBSTRING(name, $nameEnd, 1) <= '9') )";
$nameEnd = strlen($name) + 1;
$nameEndAppendix = $nameEnd + 1;
$appendix = $chunk->getAppendix();
$lenAppendix = strlen($appendix);
$checkForChunkBlob = "SUBSTRING(name, $nameEnd, $lenAppendix) = '$appendix'";
$checkForSubtableId = "(SUBSTRING(name, $nameEndAppendix, 1) >= '0'
AND SUBSTRING(name, $nameEndAppendix, 1) <= '9')";
$whereNameIs = "(name = ? OR (name LIKE ? AND ( $checkForChunkBlob OR $checkForSubtableId ) ))";
$bind = array($name, $name . '%');
} else {
if ($idSubtable === null) {
// select root table or specific record names
$bind = array_values($recordNames);
} else {
// select a subtable id
$bind = array();
foreach ($recordNames as $recordName) {
// to be backwards compatibe we need to look for the exact idSubtable blob and for the chunk
// that stores the subtables (a chunk stores many blobs in one blob)
$bind[] = $chunk->getRecordNameForTableId($recordName, $idSubtable);
$bind[] = self::appendIdSubtable($recordName, $idSubtable);
}
}
$inNames = Common::getSqlStringFieldsArray($bind);
$whereNameIs = "name IN ($inNames)";
$bind = array_values($recordNames);
}
$getValuesSql = "SELECT value, name, idsite, date1, date2, ts_archived
@ -251,110 +279,91 @@ class ArchiveSelector
if (empty($ids)) {
throw new Exception("Unexpected: id archive not found for period '$period' '");
}
// $period = "2009-01-04,2009-01-04",
$date = Date::factory(substr($period, 0, 10));
if ($archiveDataType == 'numeric') {
$isNumeric = $archiveDataType == 'numeric';
if ($isNumeric) {
$table = ArchiveTableCreator::getNumericTable($date);
} else {
$table = ArchiveTableCreator::getBlobTable($date);
}
$sql = sprintf($getValuesSql, $table, implode(',', $ids));
$sql = sprintf($getValuesSql, $table, implode(',', $ids));
$dataRows = Db::fetchAll($sql, $bind);
foreach ($dataRows as $row) {
$rows[] = $row;
if ($isNumeric) {
$rows[] = $row;
} else {
$row['value'] = self::uncompress($row['value']);
if ($chunk->isRecordNameAChunk($row['name'])) {
self::moveChunkRowToRows($rows, $row, $chunk, $loadAllSubtables, $idSubtable);
} else {
$rows[] = $row;
}
}
}
}
return $rows;
}
private static function moveChunkRowToRows(&$rows, $row, Chunk $chunk, $loadAllSubtables, $idSubtable)
{
// $blobs = array([subtableID] = [blob of subtableId])
$blobs = unserialize($row['value']);
if (!is_array($blobs)) {
return;
}
// $rawName = eg 'PluginName_ArchiveName'
$rawName = $chunk->getRecordNameWithoutChunkAppendix($row['name']);
if ($loadAllSubtables) {
foreach ($blobs as $subtableId => $blob) {
$row['value'] = $blob;
$row['name'] = self::appendIdSubtable($rawName, $subtableId);
$rows[] = $row;
}
} elseif (array_key_exists($idSubtable, $blobs)) {
$row['value'] = $blobs[$idSubtable];
$row['name'] = self::appendIdSubtable($rawName, $idSubtable);
$rows[] = $row;
}
}
public static function appendIdSubtable($recordName, $id)
{
return $recordName . "_" . $id;
}
private static function uncompress($data)
{
return @gzuncompress($data);
}
/**
* Returns the SQL condition used to find successfully completed archives that
* this instance is querying for.
*
* @param array $plugins
* @param Segment $segment
* @param bool $isSkipAggregationOfSubTables
* @return string
*/
static private function getNameCondition(array $plugins, Segment $segment, $isSkipAggregationOfSubTables)
private static function getNameCondition(array $plugins, Segment $segment)
{
// the flags used to tell how the archiving process for a specific archive was completed,
// if it was completed
$doneFlags = Rules::getDoneFlags($plugins, $segment, $isSkipAggregationOfSubTables);
$doneFlags = Rules::getDoneFlags($plugins, $segment);
$allDoneFlags = "'" . implode("','", $doneFlags) . "'";
$possibleValues = Rules::getSelectableDoneFlagValues();
// create the SQL to find archives that are DONE
return "((name IN ($allDoneFlags)) AND " .
" (value = '" . ArchiveWriter::DONE_OK . "' OR " .
" value = '" . ArchiveWriter::DONE_OK_TEMPORARY . "'))";
}
static public function purgeOutdatedArchives(Date $dateStart)
{
$purgeArchivesOlderThan = Rules::shouldPurgeOutdatedArchives($dateStart);
if (!$purgeArchivesOlderThan) {
return;
}
$idArchivesToDelete = self::getTemporaryArchiveIdsOlderThan($dateStart, $purgeArchivesOlderThan);
if (!empty($idArchivesToDelete)) {
self::deleteArchiveIds($dateStart, $idArchivesToDelete);
}
self::deleteArchivesWithPeriodRange($dateStart);
Log::debug("Purging temporary archives: done [ purged archives older than %s in %s ] [Deleted IDs: %s]",
$purgeArchivesOlderThan, $dateStart->toString("Y-m"), implode(',', $idArchivesToDelete));
}
/*
* Deleting "Custom Date Range" reports after 1 day, since they can be re-processed and would take up un-necessary space
*/
protected static function deleteArchivesWithPeriodRange(Date $date)
{
$query = "DELETE FROM %s WHERE period = ? AND ts_archived < ?";
$yesterday = Date::factory('yesterday')->getDateTime();
$bind = array(Piwik::$idPeriods['range'], $yesterday);
$numericTable = ArchiveTableCreator::getNumericTable($date);
Db::query(sprintf($query, $numericTable), $bind);
Log::debug("Purging Custom Range archives: done [ purged archives older than %s from %s / blob ]", $yesterday, $numericTable);
try {
Db::query(sprintf($query, ArchiveTableCreator::getBlobTable($date)), $bind);
} catch (Exception $e) {
// Individual blob tables could be missing
}
}
protected static function deleteArchiveIds(Date $date, $idArchivesToDelete)
{
$query = "DELETE FROM %s WHERE idarchive IN (" . implode(',', $idArchivesToDelete) . ")";
Db::query(sprintf($query, ArchiveTableCreator::getNumericTable($date)));
try {
Db::query(sprintf($query, ArchiveTableCreator::getBlobTable($date)));
} catch (Exception $e) {
// Individual blob tables could be missing
}
}
protected static function getTemporaryArchiveIdsOlderThan(Date $date, $purgeArchivesOlderThan)
{
$query = "SELECT idarchive
FROM " . ArchiveTableCreator::getNumericTable($date) . "
WHERE name LIKE 'done%'
AND (( value = " . ArchiveWriter::DONE_OK_TEMPORARY . "
AND ts_archived < ?)
OR value = " . ArchiveWriter::DONE_ERROR . ")";
$result = Db::fetchAll($query, array($purgeArchivesOlderThan));
$idArchivesToDelete = array();
if (!empty($result)) {
foreach ($result as $row) {
$idArchivesToDelete[] = $row['idarchive'];
}
}
return $idArchivesToDelete;
return "((name IN ($allDoneFlags)) AND (value IN (" . implode(',', $possibleValues) . ")))";
}
}

72
www/analytics/core/DataAccess/ArchiveTableCreator.php
View file

@ -1,6 +1,6 @@
<?php
/**
* Piwik - Open source web analytics
* Piwik - free/libre analytics platform
*
* @link http://piwik.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
@ -9,70 +9,61 @@
namespace Piwik\DataAccess;
use Exception;
use Piwik\Common;
use Piwik\Date;
use Piwik\Db;
use Piwik\DbHelper;
class ArchiveTableCreator
{
const NUMERIC_TABLE = "numeric";
const BLOB_TABLE = "blob";
const BLOB_TABLE = "blob";
public static $tablesAlreadyInstalled = null;
static public $tablesAlreadyInstalled = null;
static public function getNumericTable(Date $date)
public static function getNumericTable(Date $date)
{
return self::getTable($date, self::NUMERIC_TABLE);
}
static public function getBlobTable(Date $date)
public static function getBlobTable(Date $date)
{
return self::getTable($date, self::BLOB_TABLE);
}
static protected function getTable(Date $date, $type)
protected static function getTable(Date $date, $type)
{
$tableNamePrefix = "archive_" . $type;
$tableName = $tableNamePrefix . "_" . $date->toString('Y_m');
$tableName = $tableNamePrefix . "_" . self::getTableMonthFromDate($date);
$tableName = Common::prefixTable($tableName);
self::createArchiveTablesIfAbsent($tableName, $tableNamePrefix);
return $tableName;
}
static protected function createArchiveTablesIfAbsent($tableName, $tableNamePrefix)
protected static function createArchiveTablesIfAbsent($tableName, $tableNamePrefix)
{
if (is_null(self::$tablesAlreadyInstalled)) {
self::refreshTableList();
}
if (!in_array($tableName, self::$tablesAlreadyInstalled)) {
$db = Db::get();
$sql = DbHelper::getTableCreateSql($tableNamePrefix);
// replace table name template by real name
$tableNamePrefix = Common::prefixTable($tableNamePrefix);
$sql = str_replace($tableNamePrefix, $tableName, $sql);
try {
$db->query($sql);
} catch (Exception $e) {
// accept mysql error 1050: table already exists, throw otherwise
if (!$db->isErrNo($e, '1050')) {
throw $e;
}
}
self::getModel()->createArchiveTable($tableName, $tableNamePrefix);
self::$tablesAlreadyInstalled[] = $tableName;
}
}
static public function clear()
private static function getModel()
{
return new Model();
}
public static function clear()
{
self::$tablesAlreadyInstalled = null;
}
static public function refreshTableList($forceReload = false)
public static function refreshTableList($forceReload = false)
{
self::$tablesAlreadyInstalled = DbHelper::getTablesInstalled($forceReload);
}
@ -80,40 +71,53 @@ class ArchiveTableCreator
/**
* Returns all table names archive_*
*
* @param string $type The type of table to return. Either `self::NUMERIC_TABLE` or `self::BLOB_TABLE`.
* @return array
*/
static public function getTablesArchivesInstalled()
public static function getTablesArchivesInstalled($type = null)
{
if (is_null(self::$tablesAlreadyInstalled)) {
self::refreshTableList();
}
if (empty($type)) {
$tableMatchRegex = '/archive_(numeric|blob)_/';
} else {
$tableMatchRegex = '/archive_' . preg_quote($type) . '_/';
}
$archiveTables = array();
foreach (self::$tablesAlreadyInstalled as $table) {
if (strpos($table, 'archive_numeric_') !== false
|| strpos($table, 'archive_blob_') !== false
) {
if (preg_match($tableMatchRegex, $table)) {
$archiveTables[] = $table;
}
}
return $archiveTables;
}
static public function getDateFromTableName($tableName)
public static function getDateFromTableName($tableName)
{
$tableName = Common::unprefixTable($tableName);
$date = str_replace(array('archive_numeric_', 'archive_blob_'), '', $tableName);
$date = str_replace(array('archive_numeric_', 'archive_blob_'), '', $tableName);
return $date;
}
static public function getTypeFromTableName($tableName)
public static function getTableMonthFromDate(Date $date)
{
return $date->toString('Y_m');
}
public static function getTypeFromTableName($tableName)
{
if (strpos($tableName, 'archive_numeric_') !== false) {
return self::NUMERIC_TABLE;
}
if (strpos($tableName, 'archive_blob_') !== false) {
return self::BLOB_TABLE;
}
return false;
}
}

Some files were not shown because too many files have changed in this diff Show more