SetBreadCrumbEntry('ui-tool-bulkimport', Dict::S('Menu:CSVImportMenu'), Dict::S('UI:Title:BulkImport+'), '', 'fas fa-file-import', iTopWebPage::ENUM_BREADCRUMB_ENTRY_ICON_TYPE_CSS_CLASSES);
/**
* Helper function to build a select from the list of valid classes for a given action
*
* @deprecated 3.0.0 use GetClassesSelectUIBlock
*
* @param $sDefaultValue
* @param integer $iWidthPx The width (in pixels) of the drop-down list
* @param integer $iActionCode The ActionCode (from UserRights) to check for authorization for the classes
*
* @param string $sName The name of the select in the HTML form
*
* @return string The HTML fragment corresponding to the select tag
*/
function GetClassesSelect($sName, $sDefaultValue, $iWidthPx, $iActionCode = null)
{
DeprecatedCallsLog::NotifyDeprecatedPhpMethod('use GetClassesSelectUIBlock');
$oSelectBlock = GetClassesSelectUIBlock($sName, $sDefaultValue, $iActionCode);
return BlockRenderer::RenderBlockTemplates($oSelectBlock);
}
/**
* Helper function to build a select from the list of valid classes for a given action
*
* @param string $sName The name of the select in the HTML form
* @param $sDefaultValue
* @param integer $iWidthPx The width (in pixels) of the drop-down list
* @param integer $iActionCode The ActionCode (from UserRights) to check for authorization for the classes
*
* @return \Combodo\iTop\Application\UI\Base\Component\Input\Select\
*/
function GetClassesSelectUIBlock(string $sName, $sDefaultValue, int $iActionCode): Select
{
$oSelectBlock = SelectUIBlockFactory::MakeForSelect($sName, 'select_'.$sName);
$oOption = SelectOptionUIBlockFactory::MakeForSelectOption("", Dict::S('UI:CSVImport:ClassesSelectOne'), false);
$oSelectBlock->AddSubBlock($oOption);
$aValidClasses = array();
$aClassCategories = array('bizmodel', 'addon/authentication');
if (UserRights::IsAdministrator()) {
$aClassCategories = array('bizmodel', 'application', 'addon/authentication');
}
foreach ($aClassCategories as $sClassCategory) {
foreach (MetaModel::GetClasses($sClassCategory) as $sClassName) {
if ((is_null($iActionCode) || UserRights::IsActionAllowed($sClassName, $iActionCode)) &&
(!MetaModel::IsAbstract($sClassName))) {
$sDisplayName = MetaModel::GetName($sClassName);
$aValidClasses[$sDisplayName] = SelectOptionUIBlockFactory::MakeForSelectOption($sClassName, $sDisplayName, ($sClassName == $sDefaultValue));
}
}
}
ksort($aValidClasses);
foreach ($aValidClasses as $sValue => $oBlock) {
$oSelectBlock->AddSubBlock($oBlock);
}
return $oSelectBlock;
}
/**
* Helper to 'check' an input in an HTML form if the current value equals the value given
*
* @param mixed $sCurrentValue The current value to be chacked against the value of the input
* @param mixed $sProposedValue The value of the input
* @param bool $bInverseCondition Set to true to perform the reversed comparison
*
* @return string Either ' checked' or an empty string
*/
function IsChecked($sCurrentValue, $sProposedValue, $bInverseCondition = false)
{
$bCondition = ($sCurrentValue == $sProposedValue);
return ($bCondition xor $bInverseCondition) ? ' checked' : '';
}
/**
* Returns the number of occurences of each char from the set in the specified string
* @param string $sString The input data
* @param array $aSet The set of characters to count
* @return array 'char' => nb of occurences
*/
function CountCharsFromSet($sString, $aSet)
{
$aResult = array();
$aCount = count_chars($sString);
foreach($aSet as $sChar)
{
$aResult[$sChar] = isset($aCount[ord($sChar)]) ? $aCount[ord($sChar)] : 0;
}
return $aResult;
}
/**
* Return the most frequent (and regularly occuring) character among the given set, in the specified lines
* @param array $aCSVData The input data, one entry per line
* @param array $aPossibleSeparators The list of characters to count
* @return string The most frequent character from the set
*/
function GuessFromFrequency($aCSVData, $aPossibleSeparators)
{
$iLine = 0;
$iMaxLine = 20; // Process max 20 lines to guess the parameters
foreach($aPossibleSeparators as $sSep)
{
$aGuesses[$sSep]['total'] = $aGuesses[$sSep]['max'] = 0;
$aGuesses[$sSep]['min'] = 999;
}
$aStats = array();
while(($iLine < count($aCSVData)) && ($iLine < $iMaxLine) )
{
if (strlen($aCSVData[$iLine]) > 0)
{
$aStats[$iLine] = CountCharsFromSet($aCSVData[$iLine], $aPossibleSeparators);
}
$iLine++;
}
$iLine = 1;
foreach($aStats as $aLineStats)
{
foreach($aPossibleSeparators as $sSep)
{
$aGuesses[$sSep]['total'] += $aLineStats[$sSep];
if ($aLineStats[$sSep] > $aGuesses[$sSep]['max']) $aGuesses[$sSep]['max'] = $aLineStats[$sSep];
if ($aLineStats[$sSep] < $aGuesses[$sSep]['min']) $aGuesses[$sSep]['min'] = $aLineStats[$sSep];
}
$iLine++;
}
$aScores = array();
foreach($aGuesses as $sSep => $aData)
{
$aScores[$sSep] = $aData['total'] + $aData['max'] - $aData['min'];
}
arsort($aScores, SORT_NUMERIC); // Sort the array, higher scores first
$aKeys = array_keys($aScores);
$sSeparator = $aKeys[0]; // Take the first key, the one with the best score
return $sSeparator;
}
/**
* Try to predict the CSV parameters based on the input data
* @param string $sCSVData The input data
* @return array 'separator' => the_guessed_separator, 'qualifier' => the_guessed_text_qualifier
*/
function GuessParameters($sCSVData)
{
$aData = explode("\n", $sCSVData);
$sSeparator = GuessFromFrequency($aData, array("\t", ',', ';', '|')); // Guess the most frequent (and regular) character on each line
$sQualifier = GuessFromFrequency($aData, array('"', "'")); // Guess the most frequent (and regular) character on each line
return array('separator' => $sSeparator, 'qualifier' => $sQualifier);
}
/**
* Display a banner for the special "synchro" mode
* @param WebPage $oP The Page for the output
* @param string $sClass The class of objects to synchronize
* @param integer $iCount The number of objects to synchronize
*/
function DisplaySynchroBanner(WebPage $oP, $sClass, $iCount)
{
$oP->AddSubBlock(AlertUIBlockFactory::MakeForInformation(MetaModel::GetClassIcon($sClass)." ".Dict::Format('UI:Title:BulkSynchro_nbItem_ofClass_class', $iCount, MetaModel::GetName($sClass))));
}
/**
* Add a paragraph to the body of the page
*
* @param string $s_html
* @param ?string $sLinkUrl
*
* @return string
*/
function GetDivAlert($s_html)
{
return "
$s_html
\n";
}
/**
* Process the CSV data, for real or as a simulation
* @param WebPage $oPage The page used to display the wizard
* @param bool $bSimulate Whether or not to simulate the data load
* @return array The CSV lines in error that were rejected from the load (with the header line - if any) or null
*/
function ProcessCSVData(WebPage $oPage, $bSimulate = true)
{
$sClassName = utils::ReadParam('class_name', '', false, 'class');
// Class access right check for the import
if (UserRights::IsActionAllowed($sClassName, UR_ACTION_MODIFY) == UR_ALLOWED_NO) {
throw new CoreException(Dict::S('UI:ActionNotAllowed'));
}
// CSRF transaction id verification
if(!$bSimulate && !utils::IsTransactionValid(utils::ReadPostedParam('transaction_id', '', 'raw_data'))){
throw new CoreException(Dict::S('UI:Error:InvalidToken'));
}
$aResult = array();
$sCSVData = utils::ReadParam('csvdata', '', false, 'raw_data');
$sCSVDataTruncated = utils::ReadParam('csvdata_truncated', '', false, 'raw_data');
$sSeparator = utils::ReadParam('separator', ',', false, 'raw_data');
$sTextQualifier = utils::ReadParam('text_qualifier', '"', false, 'raw_data');
$bHeaderLine = (utils::ReadParam('header_line', '0') == 1);
$iSkippedLines = 0;
if (utils::ReadParam('box_skiplines', '0') == 1) {
$iSkippedLines = utils::ReadParam('nb_skipped_lines', '0');
}
$aFieldsMapping = utils::ReadParam('field', array(), false, 'raw_data');
$aSearchFields = utils::ReadParam('search_field', array(), false, 'field_name');
$iCurrentStep = $bSimulate ? 4 : 5;
$bAdvanced = utils::ReadParam('advanced', 0);
$sEncoding = utils::ReadParam('encoding', 'UTF-8');
$sSynchroScope = utils::ReadParam('synchro_scope', '', false, 'raw_data');
$sDateTimeFormat = utils::ReadParam('date_time_format', 'default');
$sCustomDateTimeFormat = utils::ReadParam('custom_date_time_format', (string)AttributeDateTime::GetFormat(), false, 'raw_data');
$sChosenDateFormat = ($sDateTimeFormat == 'default') ? (string)AttributeDateTime::GetFormat() : $sCustomDateTimeFormat;
if (!empty($sSynchroScope))
{
$oSearch = DBObjectSearch::FromOQL($sSynchroScope);
$sClassName = $oSearch->GetClass(); // If a synchronization scope is set, then the class is fixed !
$oSet = new DBObjectSet($oSearch);
$iCount = $oSet->Count();
DisplaySynchroBanner($oPage, $sClassName, $iCount);
$aSynchroUpdate = utils::ReadParam('synchro_update', array());
}
else
{
$sSynchroScope = '';
$aSynchroUpdate = null;
}
// Parse the data set
$oCSVParser = new CSVParser($sCSVData, $sSeparator, $sTextQualifier, MetaModel::GetConfig()->Get('max_execution_time_per_loop'));
$aData = $oCSVParser->ToArray($iSkippedLines);
$iRealSkippedLines = $iSkippedLines;
if ($bHeaderLine)
{
$aResult[] = $sTextQualifier.implode($sTextQualifier.$sSeparator.$sTextQualifier, array_shift($aData)).$sTextQualifier; // Remove the first line and store it in case of error
$iRealSkippedLines++;
}
// Format for the line numbers
$sMaxLen = (strlen(''.count($aData)) < 3) ? 3 : strlen(''.count($aData)); // Pad line numbers to the appropriate number of chars, but at least 3
// Compute the list of search/reconciliation criteria
$aSearchKeys = array();
foreach($aSearchFields as $index => $sDummy)
{
$sSearchField = $aFieldsMapping[$index];
$aMatches = array();
if (preg_match('/(.+)->(.+)/', $sSearchField, $aMatches) > 0)
{
$sSearchField = $aMatches[1];
$aSearchKeys[$aMatches[1]] = '';
}
else
{
$aSearchKeys[$sSearchField] = '';
}
if (!MetaModel::IsValidFilterCode($sClassName, $sSearchField))
{
// Remove invalid or unmapped search fields
$aSearchFields[$index] = null;
unset($aSearchKeys[$sSearchField]);
}
}
// Compute the list of fields and external keys to process
$aExtKeys = array();
$aAttributes = array();
$aExternalKeysByColumn = array();
foreach($aFieldsMapping as $iNumber => $sAttCode)
{
$iIndex = $iNumber-1;
if (!empty($sAttCode) && ($sAttCode != ':none:') && ($sAttCode != 'finalclass'))
{
if (preg_match('/(.+)->(.+)/', $sAttCode, $aMatches) > 0)
{
$sAttribute = $aMatches[1];
$sField = $aMatches[2];
$aExtKeys[$sAttribute][$sField] = $iIndex;
$aExternalKeysByColumn[$iIndex] = $sAttribute;
}
else
{
if ($sAttCode == 'id')
{
$aAttributes['id'] = $iIndex;
}
else
{
$oAttDef = MetaModel::GetAttributeDef($sClassName, $sAttCode);
if ($oAttDef->IsExternalKey())
{
$aExtKeys[$sAttCode]['id'] = $iIndex;
$aExternalKeysByColumn[$iIndex] = $sAttCode;
}
else
{
$aAttributes[$sAttCode] = $iIndex;
}
}
}
}
}
$oMyChange = null;
if (!$bSimulate)
{
// We're doing it for real, let's create a change
$sUserString = CMDBChange::GetCurrentUserName().' (CSV)';
CMDBObject::SetCurrentChangeFromParams($sUserString, CMDBChangeOrigin::CSV_INTERACTIVE);
$oMyChange = CMDBObject::GetCurrentChange();
}
$oBulk = new BulkChange(
$sClassName,
$aData,
$aAttributes,
$aExtKeys,
array_keys($aSearchKeys),
empty($sSynchroScope) ? null : $sSynchroScope,
$aSynchroUpdate,
$sChosenDateFormat, // date format
true // localize
);
$oBulk->SetReportHtml();
$oPage->AddSubBlock(InputUIBlockFactory::MakeForHidden("csvdata_truncated", $sCSVDataTruncated, "csvdata_truncated"));
$aRes = $oBulk->Process($oMyChange);
$aColumns = [];
$aColumns ["line"] = ["label" => "Line"];
$aColumns ["status"] = ["label" => "Status"];
$aColumns ["object"] = ["label" => "Object"];
foreach ($aFieldsMapping as $iNumber => $sAttCode) {
if (!empty($sAttCode) && ($sAttCode != ':none:') && ($sAttCode != 'finalclass')) {
$aColumns[$sClassName.'/'.$sAttCode] = ["label" => MetaModel::GetLabel($sClassName, $sAttCode)];
}
}
$aColumns["message"] = ["label" => "Message"];
$iErrors = 0;
$iCreated = 0;
$iModified = 0;
$iUnchanged = 0;
$aTableData = [];
foreach ($aRes as $iLine => $aResRow) {
$aTableRow = [];
$oStatus = $aResRow['__STATUS__'];
$sUrl = '';
$sMessage = '';
$sCSSRowClass = '';
$sCSSMessageClass = 'cell_ok';
switch (get_class($oStatus)) {
case 'RowStatus_NoChange':
$iUnchanged++;
$sFinalClass = $aResRow['finalclass'];
$oObj = MetaModel::GetObject($sFinalClass, $aResRow['id']->GetPureValue());
$sUrl = $oObj->GetHyperlink();
$sStatus = '';
$sCSSRowClass = 'ibo-csv-import--row-unchanged';
break;
case 'RowStatus_Modify':
$iModified++;
$sFinalClass = $aResRow['finalclass'];
$oObj = MetaModel::GetObject($sFinalClass, $aResRow['id']->GetPureValue());
$sUrl = $oObj->GetHyperlink();
$sStatus = '';
$sCSSRowClass = 'ibo-csv-import--row-modified';
break;
case 'RowStatus_Disappeared':
$iModified++;
$sFinalClass = $aResRow['finalclass'];
$oObj = MetaModel::GetObject($sFinalClass, $aResRow['id']->GetPureValue());
$sUrl = $oObj->GetHyperlink();
$sStatus = '';
$sCSSRowClass = 'ibo-csv-import--row-modified';
if ($bSimulate) {
$sMessage = Dict::S('UI:CSVReport-Object-MissingToUpdate');
} else {
$sMessage = Dict::S('UI:CSVReport-Object-MissingUpdated');
}
break;
case 'RowStatus_NewObj':
$iCreated++;
$sStatus = '';
$sCSSRowClass = 'ibo-csv-import--row-added';
if ($bSimulate) {
$sMessage = Dict::S('UI:CSVReport-Object-ToCreate');
} else {
$sFinalClass = $aResRow['finalclass'];
$oObj = MetaModel::GetObject($sFinalClass, $aResRow['id']->GetPureValue());
$sUrl = $oObj->GetHyperlink();
$sMessage = Dict::S('UI:CSVReport-Object-Created');
}
break;
case 'RowStatus_Issue':
$iErrors++;
$sMessage .= GetDivAlert($oStatus->GetDescription());
$sStatus = '
';//translate
$sCSSMessageClass = 'ibo-csv-import--cell-error';
$sCSSRowClass = 'ibo-csv-import--row-error';
if (array_key_exists($iLine, $aData)) {
$aRow = $aData[$iLine];
$aResult[] = $sTextQualifier.implode($sTextQualifier.$sSeparator.$sTextQualifier, $aRow).$sTextQualifier; // Remove the first line and store it in case of error
}
break;
}
$aTableRow['@class'] = $sCSSRowClass;
$aTableRow['line'] = sprintf("%0{$sMaxLen}d", 1 + $iLine + $iRealSkippedLines);
$aTableRow['status'] = $sStatus;
$aTableRow['object'] = $sUrl;
foreach ($aFieldsMapping as $iNumber => $sAttCode) {
if (!empty($sAttCode) && ($sAttCode != ':none:') && ($sAttCode != 'finalclass')) {
$oCellStatus = $aResRow[$iNumber - 1];
$sCellMessage = '';
if (isset($aExternalKeysByColumn[$iNumber - 1])) {
$sExtKeyName = $aExternalKeysByColumn[$iNumber - 1];
$oExtKeyCellStatus = $aResRow[$sExtKeyName];
$oExtKeyCellStatus->SetDisplayableValue($oCellStatus->GetDisplayableValue());
$oCellStatus = $oExtKeyCellStatus;
}
$sHtmlValue = $oCellStatus->GetDisplayableValue();
switch (get_class($oCellStatus)) {
case 'CellStatus_Issue':
case 'CellStatus_NullIssue':
$sCellMessage .= GetDivAlert($oCellStatus->GetDescription());
$aTableRow[$sClassName.'/'.$sAttCode] = '