Skip to Main Content

About CC-PLUS

About CC-PLUS

CC-PLUS for KYVL members

KYVL is configuring CC-PLUS to harvest institution-level COUNTER 5 reports for KYVL members from KYVL's SUSHI-compliant vendors -- Britannica, EBSCO, Gale and ProQuest.

KYVL members wishing harvest COUNTER 5 reports from other SUSHI-compliant vendors or at the school, branch or campus level may reach out to KYVL at kyvl@ky.gov.

CC-PLUS trial for other consortia

Coming soon: Request access to a CC-PLUS trial site for your consortium or institution. Email kyvl@ky.gov.

Community development

The application continues to be tested and improved. Major new features under consideration include visualization of data, creation of an API for integration with other data sources.

To join the CC-PLUS community or participate in ongoing development, contact kyvl@ky.gov.

Overview

CC-PLUS, Consortia Collaborating on a Platform for Library Usage Statistics, is a standards-based web application designed to harvest monthly usage reports in the COUNTER release 5 format from any vendor or publisher that offers such reports via a SUSHI protocol endpoint. Harvested reports are validated and aggregated into a centralized database accessible from within the application.

Reports can be customized and filtered. CC-PLUS allows users to create reports for any combination of institution, provider and date range. Custom reports can be saved as a personal views, allowing the user to return to it without having to recreate customizations.

CC-PLUS stores data on users, providers and institutions, which can be added individually in the admin interface or batch uploaded via structured CSV files. Once an institution's specific vendor credentials are configured, it's easy to create a connection and set up monthly harvesting.

Multiple consortia to be managed within a host system. The platform can also be configured for individual institutions. Following installation, additional configuration takes place within the web interface. Installation, configuration and user documentation is available as part of this guide.

CC-PLUS supports adherence to COUNTER and NISO standards in use in library, publisher, and consortial communities.

Partners and funding

This project was made possible through funding from the Institute of Museum and Library Services (IMLS) and the vision, time, and expertise of multiple consortia and volunteers across North America and Europe. Initial funding was awarded in October 2018. In January 2021, KYVL began participating in phase 2 of the ongoing CC-PLUS pilot. Recent development has been made possible through LYRASIS Catalyst Fund grant awarded to the Kentucky Virtual Library in June, 2022. For more about the history and objectives of the project, visit www.cc-plus.org.

SUSHI URL tester


Test SUSHI URL Builder

 








 

You may be taken to a new tab to view results
or a json file of results may be download to your device.

 

The test URL builder defaults to PR report and January 2023 harvest dates.
Edit the copied URL to test for other reports or dates.

INSTALLATION

CC-PLUS code

CC­-PLUS runs as a standalone web-based Laravel application connected to a MySQL database and a web server. The software is open source and is available under an Apache 2.0 software license. Download from github .

Post-installation configuration

Following installation, further configuration of global settings, consortia, institutions, providers, SUSHI connections, and users takes place within the application. Learn more at CONFIGURATION, below.

Prerequisites (2020) Minimum Version Notes
Apache 2.4+ mod_rewrite must be enabled.
MySQL  
  OR  MariaDB 
5.7.9+
10.3+
MySQL 8.x may have installation issues related to GRANT commands.
PHP 7.3+ Must include: php-gd; php-xml; php-zip; php-mysql; libapache2-mod-php.
Node.js and npm    
Composer    
Git    
DOCKER?    
     

Installation (Un*x) (DRAFT)

Step 1: Apache

Make sure you have a working apache server configured, including mod_rewrite, for serving the publicly-accessible elements of the CC-PLUS application. For the purposes of these instructions, we will refer to this place as: /var/www/ccplus/.

Define the public-facing web directory settings something along the lines of:

        DocumentRoot "/var/www/ccplus"
		. . . .
        <Directory "/var/www/ccplus">
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            allow from all
        </Directory>

Enable mod_rewrite for Apache:

# mkdir /var/www/ccplus
# a2enmod rewrite (Ubuntu, enabling for another O/S will differ)
# service apache2 restart

PHP memory settings need to be generous to support large the harvesting and decoding of large JSON reports. The initial recommendation is to set the memory limit to 1024Mb and increase from there, if necessary:

 #!  /etc/php.ini  ,   /etc/php/V.v/apache2/php.ini, or equivalent
    ...
    ; Maximum amount of memory a script may consume (128MB)
    ; http://php.net/memory-limit
    ; memory_limit = 128M
    memory_limit = 1024M

Firewalls, SSL/HTTPS, and other organizational requirements are not addressed in this document.

Step 2: Download the application

The Laravel application itself, including encryption keys, output, logs, etc. will (should) exist outside the served Apache folder. We will download the repository for the application to /usr/local and allow git to create the folder: CC-Plus.

$ cd /usr/local
$ git clone https://github.com/palcilibraries/CC-Plus.git
$ cd CC-Plus

Step 3: Set up the environment

Next, the local .env needs to be modified to match the current host environment. Use your preferred editor to open and modify the .env file:

$ cp .env.example .env
$ vi .env
  • Assign APP_URL with the URL that your webserver uses to connect to your public documents folder (step-1, above)
  • Assign database credentials (a user with rights to create databases and grant privileges):
    • DB_USERNAME
    • DB_PASSWORD
    • DB_USERNAME2
    • DB_PASSWORD2
  • Update settings for connecting to email and SMTP services (will vary depending on server environment). These settings are necessary for the system to generate forgotten-password reset links.
    • MAIL_MAILER
    • MAIL_HOST
    • MAIL_PORT
    • MAIL_USERNAME
    • MAIL_PASSWORD
    • MAIL_ENCRYPTION
    • MAIL_FROM_ADDRESS
    • MAIL_FROM_NAME

Note that the entries in .env, including the CCP_* variables at the bottom, represent default values which, if left undefined, can be overridden by values in ./config/* files.

Only two of the CCP_* settings in .env currently impact the running program (the others are for future use):

  • CCP_REPORTS : top of the path for saving raw JSON data (config-variable: reports_path)
  • MAX_HARVEST_RETRIES : limits the number of times a failed harvest will be retried (config-variable: max_harvest_retries)

CC-PLUS config variables are kept in ./config/ccplus.php. If you want to clear any CCP_* values in the .env file, be aware that the config file values would then be applied. To confirm their values or to modify them, edit the file:

$ vi ./config/ccplus.php

If CCP_REPORTS (in .env) or ‘reports_path’ (in config/ccplus.php) point to a folder, make sure it exists, and is writeable by the webserver. For example:

# cd /usr/local
# mkdir stats_reports
# chown root:www-data stats_reports
# chmod g+rw stats_reports

Step 4: Install the application

First we will setup the application Kernel file: For simplicity sake, we’ll configure our initial installation a single consortium. You can edit the Kernel.php file to match your operational needs, especially as they relate to automating report harvesting. You don’t need to define the schedule at this point, but doing so now won’t hurt anything.

$ cd /usr/local/CC-Plus/app/Console
$ cp Kernel.php.example-single ./Kernel.php
$ cd ../..

Now run the composer install:

$ composer install

And then install npm:

$ npm install

You also need to generate an encryption key for the application. This key will be used to encrypt the raw JSON report and application data, not passwords. This only needs to be done during installation. Resetting this key later will make any existing saved data unrecoverable unless you maintain a record of all previous key value(s). This command will update the .env with a unique value for APP_KEY.

$ php artisan key:generate
   Application key set successfully

Next run npm to build the application and the publicly-accessible files for the webserver

$ npm run prod

The webserver will need to be able to write to some folders within the application folder. Assuming the webserver executes with group-membership for www-data :

# cd /usr/local/CC-Plus/
# chown -R root:www-data storage
# chmod -R g+rw storage
# chown root:www-data bootstrap/cache
# chmod g+rw bootstrap/cache

Step 5: Update the webserver directory

Setup public/index.php to reflect the installation path for the application. If you are installing to a location other than /usr/local/CC-Plus, then you’ll need to change the value of the _CCPHOME_ variable to match your installation path:

$ cd /usr/local/CC-Plus/public/
$ mv index.php.example ./index.php
$ vi index.php
   . . . .
	define('_CCPHOME_','/usr/local/CC-Plus/');  // Modify this line as necessary, and include a trailing slash
   . . . .
$ cd ..

Then copy the publicly accessible files to the public webserver folder:

# cp -r /usr/local/CC-Plus/public/. /var/www/ccplus/
# chown -R root:www-data /var/www/ccplus

Step 6: Set up initial databases

Begin this step by creating the two initial CC-Plus databases (using the same user defined in step-3 above):

$ mysql
mysql> create database ccplus_global;
mysql> create database ccplus_con_template;
mysql> quit

Step 7: Migrate initial database tables

The tables in the ccplus_global database will be shared by all consortia within the host system

$ cd /usr/local/CC-Plus
$ php artisan migrate:fresh --database=globaldb --path=database/migrations/global
Dropped all tables successfully
Migration table created successfully
Migrating: 2019_07_12_200315_create_datatypes_table
 . . .
Migrated: 2020_01_15_153540_create_jobs_table (108.75ms)
$

The tables in the ccplus_con_template database are used when creating consortia for CC-Plus

$ php artisan migrate:fresh --database=con_template --path=database/migrations/con_template
Dropped all tables successfully
Migration table created successfully
Migrating: 2019_07_16_111258_create_institutiontypes_table
 . . .
Migrated: 2020_05_29_133354_create_system_alerts_table (89.93ms)
$

Step 8: Seed tables

Certain tables in both the global and the template need to be seeded with some initial data.

$ php artisan db:seed
Seeding: Database\Seeders\ReportsTableSeeder
 . . .
Seeded: Database\Seeders\CcplusErrorsTableSeeder (10.23ms)
$

Step 9: Add a consortium

The ccplus:add_consortium command script prompts for inputs and creates the new consortium. Note: The “database key” is used to create a consortium-specific database named “ccplus_< database-key-value >”.

$ php artisan ccplus:addconsortium
  New consortium name?:
  > MyConsortium

  Primary email for the consortium?:
  > my.email@some.domain.com

  Provide a unique database key for the consortium
   (default creates a random string) []:
  > MyCon1

  Make it active (Y/N) [Y]?:
  > Y

Dropped all tables successfully.
Migration table created successfully.
Migrating .....
  .  .  .  .
Migrated .....
New database migration completed with status: 0
Seeding .....
  .  .  .  .
Seeded: .....
Database seeding completed successfully
Initial database seeding completed with status: 0
Consortium added to global database.
The initial Administrator account for a new consortium is always created with
an email address set to "Administrator".

 Enter a password for this Administrator account?:
 > MyAdminPass

New consortium: MyConsortium Successfully Created.
NOTE: app/Console/Kernel.php needs updating in order to automate harvesting!
$

Congratulations!

You should now be able to connect and login to the application using the Administrator credential for your initial consortium! You can now create users, institutions, and providers through the web interface.

Step 10: Define harvesting schedule (optional)

Automated harvesting for CC-Plus is defined using the schedule defined in app/Console/Kernel.php (which we created in Step 4, above). The initial file is configured to automate harvesting for a single consortium using two queue handler processes (workers) which are scheduled to run every ten minutes. This means that at least one of the workers will wake and check for recently queued jobs every 10-minutes. An example file for a two-consortium configuration is also included, named: Kernel.php.example-multiple.

More details on scheduling tasks in Laravel applications can be found at laravel.com/docs/8.x/scheduling.

Step 11: Add scheduler to system cron (optional)

The default Kernel.php Scheduler configuration expects to be launched on a regular interval (for example, every 10 minutes). If nothing needs to be processed, the scheduler will exit until the next cycle. These lines (or a close approximation) need to be added to the system cron processing to enable unattended harvesting:

# Run CC+ Laravel scheduler every 10 minutes
*/10 * * * * root cd /usr/local/CC-Plus && /usr/bin/php /usr/local/CC-Plus/artisan schedule:run >> /dev/null 2>&1

Artisan commands (DRAFT)

CC-PLUS Artisan Commands

The Laravel environment for CC-Plus includes a set of system-level console commands to manage or operate parts of the application. A list of the commands can be displayed via:

$ cd /usr/local/CC-Plus
$ php artisan | grep ccplus

Help for the individual commands is also available, for example:

$ cd /usr/local/CC-Plus
$ php artisan help ccplus:addconsortium

A brief description of each command is below.

See the help screen for each command for complete details on arguments and options.

ccplus:addconsortium Adds a database and administrator credential to a CC-Plus host system.
ccplus:data-archive Exports stored CC-PLUS report data, institution/provider configuration, and, optionally, global table data, to an importable .SQL file.
ccplus:data-purge Removes stored CC-PLUS report data from the database.
ccplus:sushibatch Command-line submission to batch-process the submission of report harvests.
ccplus:sushiloader Intended to run nightly by the Kernel.php scheduler, this command scans the SUSHI settings for all institutions and providers within a consortium and loads requests into the global jobs queue (globaldb:jobs table).
ccplus:sushiqw Intended to run by the Kernel.php scheduler (by default every 10 minutes), this command scans the jobs queue, issues SUSHI requests to report providers, and stores/logs the results.
ccplus:C5test This is a command for testing raw report data. Accepts COUNTER-5 JSON report data from a file and attempts to validate and store it in the running system.
  Reset Global Admin password

CONFIGURATION

Initial Setup

Global admin

 A user with the global admin role is created during the installation process (see step #).

Initial setup

Initial configuration of a working install requires setup of:

  • One or more consortia
  • Providers
  • Institutions (and, optionally, institution groups)
  • Users
  • SUSHI connections

Details regarding initial setup and ongoing maintenance of each of these entities appears below.

Only a global admin can add a consortium. The global admin can then configure or delegate further configuration of consortia, institutions, providers, SUSHI connections and users to consortial admin(s) and local (institutional) admins.

Global settings (DRAFT)

To access global settings, log in to your CC-PLUS application using the global admin credentials created at the time of installation by your sysadmin.

Add a consortial instance

In the Consortial Instances section, click Add Consortium. You will be asked to provide:

  1. Consortium name
  2. Database prefix key. Caution: key cannot be modified in the application.
  3. Administrator password for default Administrator
  4. Email address of consortium administrator.

By default, the new consortium will be set to active.

Configure global settings

In the Global Settings section, review and edit the following settings. These settings apply to all consortia created in this installation.

Setting Description Default Edit or view?
Root URL (define) http://localhost Edit
Reports path (define) /usr/local/stats_reports Edit
Cookie life (define)   Edit
Debug SQL queries (define)   Edit
Log loin fails (define)   Edit
Max harvest retries (define)   Edit
Max name length (define)   Edit
Silence days (define)   Edit

Mail settings must be configured on the server to enable Forgot Password functionality. Your sysadmin will be asked to provide:

  1. Mail protocol (required) - e.g., IMAP, POP3, SMTP.
  2. Encryption (required) - default is none.
  3. Port number (required)- e.g., 25, 587, 465, 2525.
  4. Username (optional)
  5. Password (optional)

Configure global providers

A comprehensive list of known COUNTER-compliant providers [link to file / API?], is available for import into the CC-PLUS web application at the time of initial configuration. Each provider is configured with SUSHI service URL, required credentials and available reports to simplify subsequent configuration of SUSHI connections for individual institutions. By default, all providers are set to active.

Provider are available across all consortia and institutions. Provider definitions are authoritative and can be created and maintained by the global admin only and serve all consortia on the application. Local and consortial admins may make changes to optional settings, such as which available reports to harvest monthly harvest day, but cannot change provider definitions. Individual SUSHI connections may be made by global, consortial or local (institutional) admins.

Consortial admins may duplicate a database to restrict changes by local admins (?). 

Provider configuration requires:

  1. Provider name
  2. SUSHI service URL
  3. Required connection fields
  4. Available reports

Return to this section to make any updates or to suspend providers across all consortia. Changes affect all SUSHI connections to updated providers in all consortial instances.

Global Providers

Col Column Name Required Description Data Type Valid Values Default
A Provider ID Yes Unique system ID Integer Unique values only  
B

Provider Name

Yes Provider name String    
C SUSHI Server URL Yes URL for provider SUSHI service String Valid URL  
D DR   Is database report available? String Y or N N
E IR   Is item report available? String Y or N N
F PR   Is platform report available? String Y or N N
G TR   Is title report available? String Y or N N
H Customer ID   Required by provider? String Y or N N
I Requestor ID   Required by provider? String Y or N N
J API Key   Required by provider? String Y or N N
K Custom Parameters   Required by provider? String Y or N N
L Custom Parameters Pattern   Custom parameter-value pairs each beginning with "&" to serve as help text for adding institutional credentials, e.g.,
&requiredCredentialName=value&param2=value
String   (Ignored if Custom Parameters is "N")
M Source URL   Link to authority for supplied information String Valid URL  
N ConfirmedDate   Date Source URL accessed OR last successful harvest date Date   Today's date
O Active   Set provider to active? String Y or N Y

Optional configuration for individual institutions or consortia includes:

  • Harvest date
  • Max harvest retries.

Both of these settings have a global default. A consortial admin or local (institutional) admin may choose to override these values for all institutional or consortial providers or individually by provider.

Institution Groups (DRAFT)

Institution groups may be configured by a consortial admin or by a global admin upon selecting the appropriate consortial instance.

Configuring institution groups

  • Use of institution groups is optional.
  • If you wish to use institution groups, recommended procedure is to configure them before importing institutions.
  • Institution groups can be bulk imported or configured manually within the application. Import is recommended for large data sets.
  • Institution groups are found in the Reports menu.

Import steps

  • Begin with a full export of institution groups.
  • Update the data sheet as specified in the table below.
  • Save sheet as CSV UTF-8 for import.
  • Return to institution groups page to import.

Notes

  1. Imports support additions and updates of institution groups. They cannot be used to delete institution groups. 
  2. Rows lacking required values will be ignored.
  3. Invalid values that are not required will be set to default value.
  4. Data in any columns beyond column B will be ignored.

Import format for institution groups

Col Column Name Required? Data Type Default Description Notes
A InstitutionGroupID Yes Integer
(unique)
  Unique system ID

Incremented from previous high value.

B InstitutionGroupName Yes String   Institution name  

Institutions (DRAFT)

Institutions may be configured by a consortial admin or by a global admin upon selecting the appropriate consortial instance.

Users with a local admin role may make some changes to their local institution. 

Configuring institutions

  • Use of institution groups is optional.
  • If you wish to use institution groups, recommended procedure is to configure them before importing institutions.
  • Institutions can be bulk imported or configured manually within the application. Import is recommended for large data sets.
  • Institutions are found in the Admin menu.

Import steps

  • Begin with a full export of institutions.
  • Update the data sheet as specified in the table below.
  • Save sheet as CSV UTF-8 for import.
  • Return to institutions page to import.

Notes

  1. Imports support additions and updates of institutions. They cannot be used to delete institutions. 
  2. Rows lacking required values will be ignored.
  3. Invalid values that are not required will be set to default value.
  4. Data in any columns beyond column G will be ignored.
  5. InstitutionID values (column A) take precedence over LocalID values (column B) when processing import records. If a match is found for column A, all other values in the row are treated as updates.

Import format for institutions

Col Column Name Required? Data Type Default Description Notes
A InstitutionID Yes Integer >1
(unique)
. Unique system ID ID=1 reserved for system use.
Incremented from previous high value
B LocalID No String NULL    
C InstitutionName Yes String   Institution name  
D Active? No String
(Y or N)
Y Make the institution active?  
E FTE No Integer NULL Institutional FTE  
F Group(s) No Comma-separated integers     Uses InstitutionGroupID values from Institution groups
G Notes No Text blob NULL Notes or other details  

Users (DRAFT)

Users may be configured by users with global admin or consortial admin roles, and are accessible from the Admin menu.

A user may make limited changes to his or her individual profile. 

User types

User types

Role Who Permissions Notes
GlobalAdmin Global admin   Created during installation process.
ConsortialAdmin Consortium Staff Add providers, add institutions, run consortial reports, import data  
Local admin Local systems staff Connect to providers, run local reports  
Viewer Most consortial users? View consortial data  
User View local stats only View local data  

Users

  • The template above is for reference only. To add to or make changes to existing data, download the Users template at CC-PLUS > Settings > Users.
  • Edit the Users tab on the downloaded spreadsheet to import or update user settings.
    • When performing full-replacement imports, be VERY careful about changing or overwriting existing ID value(s).
    • The best approach is to add to or modify a full export to ensure that existing user IDs are not accidently overwritten.
  • Once edits are complete, save the spreadsheet as a CSV and import it into CC-Plus on the same page from which the template was downloaded.

The table below describes the datatype and order that the import expects.

  • Any Import rows without an ID value in column 'A' will be ignored.
  • If values are missing/invalid for a column, but not required, they will be set to the 'Default'.
  • Any header row or columns beyond 'H' will be ignored.
Col Column Name Data Type Description Required? Unique? Default
A Id Integer CC-PLUS User ID Yes Yes  
B Email String Email address Yes Yes  
C Password String Password (will be encrypted)     NULL - no change
D Name String Full name     NULL
E Phone String Phone number     NULL
F Active String (Y or N) Make the user active?     Y
G Role(s) Comma-separated strings Admin, Manager, User or Viewer (see below)     User
H Institution ID Integer CC-PLUS Institution ID Yes   1 (consortium staff)

User types

Role Who Permissions Examples
*Admin Consortium Staff Add providers, add institutions, run consortial reports, import data KYVL, KCTCS, JCPS
Manager Local systems staff Connect to providers, run local reports  
Viewer Most consortial users? View consortial data  
User View local stats only View local data  

*Admins have the ability to import spreadsheets. PLEASE DO NOT IMPORT without contacting KYVL admin. DO NOT select Full Replacement when importing. See more information below.