KYVL is configuring CC-PLUS to harvest institution-level COUNTER 5 reports for KYVL members from KYVL's SUSHI-compliant vendors -- Britannica, EBSCO, Gale and ProQuest.
KYVL members wishing harvest COUNTER 5 reports from other SUSHI-compliant vendors or at the school, branch or campus level may reach out to KYVL at kyvl@ky.gov.
Coming soon: Request access to a CC-PLUS trial site for your consortium or institution. Email kyvl@ky.gov.
The application continues to be tested and improved. Major new features under consideration include visualization of data, creation of an API for integration with other data sources.
To join the CC-PLUS community or participate in ongoing development, contact kyvl@ky.gov.
CC-PLUS, Consortia Collaborating on a Platform for Library Usage Statistics, is a standards-based web application designed to harvest monthly usage reports in the COUNTER release 5 format from any vendor or publisher that offers such reports via a SUSHI protocol endpoint. Harvested reports are validated and aggregated into a centralized database accessible from within the application.
Reports can be customized and filtered. CC-PLUS allows users to create reports for any combination of institution, provider and date range. Custom reports can be saved as a personal views, allowing the user to return to it without having to recreate customizations.
CC-PLUS stores data on users, providers and institutions, which can be added individually in the admin interface or batch uploaded via structured CSV files. Once an institution's specific vendor credentials are configured, it's easy to create a connection and set up monthly harvesting.
Multiple consortia to be managed within a host system. The platform can also be configured for individual institutions. Following installation, additional configuration takes place within the web interface. Installation, configuration and user documentation is available as part of this guide.
CC-PLUS supports adherence to COUNTER and NISO standards in use in library, publisher, and consortial communities.
This project was made possible through funding from the Institute of Museum and Library Services (IMLS) and the vision, time, and expertise of multiple consortia and volunteers across North America and Europe. Initial funding was awarded in October 2018. In January 2021, KYVL began participating in phase 2 of the ongoing CC-PLUS pilot. Recent development has been made possible through LYRASIS Catalyst Fund grant awarded to the Kentucky Virtual Library in June, 2022. For more about the history and objectives of the project, visit www.cc-plus.org.
CC-PLUS runs as a standalone web-based Laravel application connected to a MySQL database and a web server. The software is open source and is available under an Apache 2.0 software license. Download from github .
Following installation, further configuration of global settings, consortia, institutions, providers, SUSHI connections, and users takes place within the application. Learn more at CONFIGURATION, below.
Installation (Un*x) steps
Creation of global admin
Make sure you have a working apache server configured, including mod_rewrite
, for serving the publicly-accessible elements of the CC-PLUS application. For the purposes of these instructions, we will refer to this place as: /var/www/ccplus/
.
Define the public-facing web directory settings something along the lines of:
DocumentRoot "/var/www/ccplus"
. . . .
<Directory "/var/www/ccplus">
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</Directory>
Enable mod_rewrite for Apache:
# mkdir /var/www/ccplus
# a2enmod rewrite (Ubuntu, enabling for another O/S will differ)
# service apache2 restart
PHP memory settings need to be generous to support large the harvesting and decoding of large JSON reports. The initial recommendation is to set the memory limit to 1024Mb and increase from there, if necessary:
#! /etc/php.ini , /etc/php/V.v/apache2/php.ini, or equivalent
...
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
; memory_limit = 128M
memory_limit = 1024M
Firewalls, SSL/HTTPS, and other organizational requirements are not addressed in this document.
The Laravel application itself, including encryption keys, output, logs, etc. will (should) exist outside the served Apache folder. We will download the repository for the application to /usr/local
and allow git
to create the folder: CC-Plus
.
$ cd /usr/local
$ git clone https://github.com/palcilibraries/CC-Plus.git
$ cd CC-Plus
Next, the local .env
needs to be modified to match the current host environment. Use your preferred editor to open and modify the .env file:
$ cp .env.example .env
$ vi .env
Note that the entries in .env
, including the CCP_*
variables at the bottom, represent default values which, if left undefined, can be overridden by values in ./config/*
files.
Only two of the CCP_*
settings in .env
currently impact the running program (the others are for future use):
CC-PLUS config variables are kept in ./config/ccplus.php
. If you want to clear any CCP_*
values in the .env
file, be aware that the config file values would then be applied. To confirm their values or to modify them, edit the file:
$ vi ./config/ccplus.php
If CCP_REPORTS (in .env) or ‘reports_path’ (in config/ccplus.php) point to a folder, make sure it exists, and is writeable by the webserver. For example:
# cd /usr/local
# mkdir stats_reports
# chown root:www-data stats_reports
# chmod g+rw stats_reports
First we will setup the application Kernel file: For simplicity sake, we’ll configure our initial installation a single consortium. You can edit the Kernel.php file to match your operational needs, especially as they relate to automating report harvesting. You don’t need to define the schedule at this point, but doing so now won’t hurt anything.
$ cd /usr/local/CC-Plus/app/Console
$ cp Kernel.php.example-single ./Kernel.php
$ cd ../..
Now run the composer install:
$ composer install
And then install npm:
$ npm install
You also need to generate an encryption key for the application. This key will be used to encrypt the raw JSON report and application data, not passwords. This only needs to be done during installation. Resetting this key later will make any existing saved data unrecoverable unless you maintain a record of all previous key value(s). This command will update the .env
with a unique value for APP_KEY.
$ php artisan key:generate
Application key set successfully
Next run npm to build the application and the publicly-accessible files for the webserver
$ npm run prod
The webserver will need to be able to write to some folders within the application folder. Assuming the webserver executes with group-membership for www-data
:
# cd /usr/local/CC-Plus/
# chown -R root:www-data storage
# chmod -R g+rw storage
# chown root:www-data bootstrap/cache
# chmod g+rw bootstrap/cache
Setup public/index.php
to reflect the installation path for the application. If you are installing to a location other than /usr/local/CC-Plus
, then you’ll need to change the value of the _CCPHOME_
variable to match your installation path:
$ cd /usr/local/CC-Plus/public/
$ mv index.php.example ./index.php
$ vi index.php
. . . .
define('_CCPHOME_','/usr/local/CC-Plus/'); // Modify this line as necessary, and include a trailing slash
. . . .
$ cd ..
Then copy the publicly accessible files to the public webserver folder:
# cp -r /usr/local/CC-Plus/public/. /var/www/ccplus/
# chown -R root:www-data /var/www/ccplus
Begin this step by creating the two initial CC-Plus databases (using the same user defined in step-3 above):
$ mysql
mysql> create database ccplus_global;
mysql> create database ccplus_con_template;
mysql> quit
The tables in the ccplus_global database will be shared by all consortia within the host system
$ cd /usr/local/CC-Plus
$ php artisan migrate:fresh --database=globaldb --path=database/migrations/global
Dropped all tables successfully
Migration table created successfully
Migrating: 2019_07_12_200315_create_datatypes_table
. . .
Migrated: 2020_01_15_153540_create_jobs_table (108.75ms)
$
The tables in the ccplus_con_template database are used when creating consortia for CC-Plus
$ php artisan migrate:fresh --database=con_template --path=database/migrations/con_template
Dropped all tables successfully
Migration table created successfully
Migrating: 2019_07_16_111258_create_institutiontypes_table
. . .
Migrated: 2020_05_29_133354_create_system_alerts_table (89.93ms)
$
Certain tables in both the global and the template need to be seeded with some initial data.
$ php artisan db:seed
Seeding: Database\Seeders\ReportsTableSeeder
. . .
Seeded: Database\Seeders\CcplusErrorsTableSeeder (10.23ms)
$
The ccplus:add_consortium
command script prompts for inputs and creates the new consortium. Note: The “database key” is used to create a consortium-specific database named “ccplus_< database-key-value >”.
$ php artisan ccplus:addconsortium
New consortium name?:
> MyConsortium
Primary email for the consortium?:
> my.email@some.domain.com
Provide a unique database key for the consortium
(default creates a random string) []:
> MyCon1
Make it active (Y/N) [Y]?:
> Y
Dropped all tables successfully.
Migration table created successfully.
Migrating .....
. . . .
Migrated .....
New database migration completed with status: 0
Seeding .....
. . . .
Seeded: .....
Database seeding completed successfully
Initial database seeding completed with status: 0
Consortium added to global database.
The initial Administrator account for a new consortium is always created with
an email address set to "Administrator".
Enter a password for this Administrator account?:
> MyAdminPass
New consortium: MyConsortium Successfully Created.
NOTE: app/Console/Kernel.php needs updating in order to automate harvesting!
$
Congratulations!
You should now be able to connect and login to the application using the Administrator credential for your initial consortium! You can now create users, institutions, and providers through the web interface.
Automated harvesting for CC-Plus is defined using the schedule defined in app/Console/Kernel.php
(which we created in Step 4, above). The initial file is configured to automate harvesting for a single consortium using two queue handler processes (workers) which are scheduled to run every ten minutes. This means that at least one of the workers will wake and check for recently queued jobs every 10-minutes. An example file for a two-consortium configuration is also included, named: Kernel.php.example-multiple
.
More details on scheduling tasks in Laravel applications can be found at laravel.com/docs/8.x/scheduling.
The default Kernel.php Scheduler configuration expects to be launched on a regular interval (for example, every 10 minutes). If nothing needs to be processed, the scheduler will exit until the next cycle. These lines (or a close approximation) need to be added to the system cron processing to enable unattended harvesting:
# Run CC+ Laravel scheduler every 10 minutes
*/10 * * * * root cd /usr/local/CC-Plus && /usr/bin/php /usr/local/CC-Plus/artisan schedule:run >> /dev/null 2>&1
The Laravel environment for CC-Plus includes a set of system-level console commands to manage or operate parts of the application. A list of the commands can be displayed via:
$ cd /usr/local/CC-Plus
$ php artisan | grep ccplus
Help for the individual commands is also available, for example:
$ cd /usr/local/CC-Plus
$ php artisan help ccplus:addconsortium
A brief description of each command is below.
See the help screen for each command for complete details on arguments and options.
ccplus:addconsortium |
Adds a database and administrator credential to a CC-Plus host system. |
ccplus:data-archive |
Exports stored CC-PLUS report data, institution/provider configuration, and, optionally, global table data, to an importable .SQL file. |
ccplus:data-purge |
Removes stored CC-PLUS report data from the database. |
ccplus:sushibatch |
Command-line submission to batch-process the submission of report harvests. |
ccplus:sushiloader |
Intended to run nightly by the Kernel.php scheduler, this command scans the SUSHI settings for all institutions and providers within a consortium and loads requests into the global jobs queue (globaldb:jobs table). |
ccplus:sushiqw |
Intended to run by the Kernel.php scheduler (by default every 10 minutes), this command scans the jobs queue, issues SUSHI requests to report providers, and stores/logs the results. |
ccplus:C5test |
This is a command for testing raw report data. Accepts COUNTER-5 JSON report data from a file and attempts to validate and store it in the running system. |
Reset Global Admin password |
A user with the global admin role is created during the installation process (see step #).
Initial configuration of a working install requires setup of:
Details regarding initial setup and ongoing maintenance of each of these entities appears below.
Only a global admin can add a consortium. The global admin can then configure or delegate further configuration of consortia, institutions, providers, SUSHI connections and users to consortial admin(s) and local (institutional) admins.
To access global settings, log in to your CC-PLUS application using the global admin credentials created at the time of installation by your sysadmin.
In the Consortial Instances section, click Add Consortium. You will be asked to provide:
By default, the new consortium will be set to active.
In the Global Settings section, review and edit the following settings. These settings apply to all consortia created in this installation.
Setting | Description | Default | Edit or view? |
---|---|---|---|
Root URL | (define) | http://localhost | Edit |
Reports path | (define) | /usr/local/stats_reports | Edit |
Cookie life | (define) | Edit | |
Debug SQL queries | (define) | Edit | |
Log loin fails | (define) | Edit | |
Max harvest retries | (define) | Edit | |
Max name length | (define) | Edit | |
Silence days | (define) | Edit |
Mail settings must be configured on the server to enable Forgot Password functionality. Your sysadmin will be asked to provide:
A comprehensive list of known COUNTER-compliant providers [link to file / API?], is available for import into the CC-PLUS web application at the time of initial configuration. Each provider is configured with SUSHI service URL, required credentials and available reports to simplify subsequent configuration of SUSHI connections for individual institutions. By default, all providers are set to active.
Provider are available across all consortia and institutions. Provider definitions are authoritative and can be created and maintained by the global admin only and serve all consortia on the application. Local and consortial admins may make changes to optional settings, such as which available reports to harvest monthly harvest day, but cannot change provider definitions. Individual SUSHI connections may be made by global, consortial or local (institutional) admins.
Consortial admins may duplicate a database to restrict changes by local admins (?).
Provider configuration requires:
Return to this section to make any updates or to suspend providers across all consortia. Changes affect all SUSHI connections to updated providers in all consortial instances.
The table below describes the datatype and order that the import expects.
Col | Column Name | Required | Description | Data Type | Valid Values | Default |
---|---|---|---|---|---|---|
A | Provider ID | Yes | Unique system ID | Integer | Unique values only | |
B |
Provider Name |
Yes | Provider name | String | ||
C | SUSHI Server URL | Yes | URL for provider SUSHI service | String | Valid URL | |
D | DR | Is database report available? | String | Y or N | N | |
E | IR | Is item report available? | String | Y or N | N | |
F | PR | Is platform report available? | String | Y or N | N | |
G | TR | Is title report available? | String | Y or N | N | |
H | Customer ID | Required by provider? | String | Y or N | N | |
I | Requestor ID | Required by provider? | String | Y or N | N | |
J | API Key | Required by provider? | String | Y or N | N | |
K | Custom Parameters | Required by provider? | String | Y or N | N | |
L | Custom Parameters Pattern | Custom parameter-value pairs each beginning with "&" to serve as help text for adding institutional credentials, e.g., &requiredCredentialName=value¶m2=value |
String | (Ignored if Custom Parameters is "N") | ||
M | Source URL | Link to authority for supplied information | String | Valid URL | ||
N | ConfirmedDate | Date Source URL accessed OR last successful harvest date | Date | Today's date | ||
O | Active | Set provider to active? | String | Y or N | Y |
Optional configuration for individual institutions or consortia includes:
Both of these settings have a global default. A consortial admin or local (institutional) admin may choose to override these values for all institutional or consortial providers or individually by provider.
Institution groups may be configured by a consortial admin or by a global admin upon selecting the appropriate consortial instance.
Import steps
Notes
Col | Column Name | Required? | Data Type | Default | Description | Notes |
---|---|---|---|---|---|---|
A | InstitutionGroupID | Yes | Integer (unique) |
Unique system ID |
Incremented from previous high value. |
|
B | InstitutionGroupName | Yes | String | Institution name |
Institutions may be configured by a consortial admin or by a global admin upon selecting the appropriate consortial instance.
Users with a local admin role may make some changes to their local institution.
Import steps
Notes
Col | Column Name | Required? | Data Type | Default | Description | Notes |
---|---|---|---|---|---|---|
A | InstitutionID | Yes | Integer >1 (unique) |
. | Unique system ID | ID=1 reserved for system use. Incremented from previous high value |
B | LocalID | No | String | NULL | ||
C | InstitutionName | Yes | String | Institution name | ||
D | Active? | No | String (Y or N) |
Y | Make the institution active? | |
E | FTE | No | Integer | NULL | Institutional FTE | |
F | Group(s) | No | Comma-separated integers | Uses InstitutionGroupID values from Institution groups | ||
G | Notes | No | Text blob | NULL | Notes or other details |
Users may be configured by users with global admin or consortial admin roles, and are accessible from the Admin menu.
A user may make limited changes to his or her individual profile.
User types
Role | Who | Permissions | Notes |
---|---|---|---|
GlobalAdmin | Global admin | Created during installation process. | |
ConsortialAdmin | Consortium Staff | Add providers, add institutions, run consortial reports, import data | |
Local admin | Local systems staff | Connect to providers, run local reports | |
Viewer | Most consortial users? | View consortial data | |
User | View local stats only | View local data |
The table below describes the datatype and order that the import expects.
Col | Column Name | Data Type | Description | Required? | Unique? | Default |
---|---|---|---|---|---|---|
A | Id | Integer | CC-PLUS User ID | Yes | Yes | |
B | String | Email address | Yes | Yes | ||
C | Password | String | Password (will be encrypted) | NULL - no change | ||
D | Name | String | Full name | NULL | ||
E | Phone | String | Phone number | NULL | ||
F | Active | String (Y or N) | Make the user active? | Y | ||
G | Role(s) | Comma-separated strings | Admin, Manager, User or Viewer (see below) | User | ||
H | Institution ID | Integer | CC-PLUS Institution ID | Yes | 1 (consortium staff) |
User types
Role | Who | Permissions | Examples |
---|---|---|---|
*Admin | Consortium Staff | Add providers, add institutions, run consortial reports, import data | KYVL, KCTCS, JCPS |
Manager | Local systems staff | Connect to providers, run local reports | |
Viewer | Most consortial users? | View consortial data | |
User | View local stats only | View local data |
*Admins have the ability to import spreadsheets. PLEASE DO NOT IMPORT without contacting KYVL admin. DO NOT select Full Replacement when importing. See more information below.